[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:27:58 dist-test-slave-p7jt krb5kdc[8318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:27:58 dist-test-slave-p7jt krb5kdc[8318](info): set up 2 sockets
Sep 23 17:27:58 dist-test-slave-p7jt krb5kdc[8318](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 17:27:59.534583 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.012s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:27:59 dist-test-slave-p7jt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:27:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:27:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:27:59.704634 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:38833
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:38833
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:00.031867 8331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:00.032215 8331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:00.032323 8331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:00.039805 8331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:00.039924 8331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:00.039975 8331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:00.040055 8331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:00.040104 8331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:00.050880 8331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:38833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:38833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:00.052489 8331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:00.053995 8331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:00.066787 8337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:00.066854 8336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.067039 8331 server_base.cc:1034] running on GCE node
W20260923 17:28:00.067339 8339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.068157 8331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:00.069684 8331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:00.070935 8331 hybrid_clock.cc:648] HybridClock initialized: now 1790184480070838 us; error 47 us; skew 500 ppm
I20260923 17:28:00.071301 8331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:00 dist-test-slave-p7jt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:00.074545 8331 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:00.076308 8331 webserver.cc:460] Webserver started at http://127.8.29.254:33203/ using document root <none> and password file <none>
I20260923 17:28:00.077031 8331 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:00.077152 8331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:00.077457 8331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:00.080830 8331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "6f485319943b42309a63d15c20f0397c"
format_stamp: "Formatted at 2026-09-23 17:28:00 on dist-test-slave-p7jt"
server_key: "98d618942c8cd2b102b9942c88110e49"
server_key_iv: "41ea9e33d41745e969689c7073b93207"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.081605 8331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "6f485319943b42309a63d15c20f0397c"
format_stamp: "Formatted at 2026-09-23 17:28:00 on dist-test-slave-p7jt"
server_key: "98d618942c8cd2b102b9942c88110e49"
server_key_iv: "41ea9e33d41745e969689c7073b93207"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.087194 8331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:28:00.090919 8346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.092377 8331 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:00.092557 8331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "6f485319943b42309a63d15c20f0397c"
format_stamp: "Formatted at 2026-09-23 17:28:00 on dist-test-slave-p7jt"
server_key: "98d618942c8cd2b102b9942c88110e49"
server_key_iv: "41ea9e33d41745e969689c7073b93207"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.092770 8331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:00.116616 8331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:00.121151 8331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:00.121430 8331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:00.140758 8331 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:38833
I20260923 17:28:00.140805 8397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:38833 every 8 connection(s)
I20260923 17:28:00.142225 8331 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:00.146113 8398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:00.151786 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 8331
I20260923 17:28:00.152076 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:00.153959 8311 external_mini_cluster.cc:1428] Setting key b2fc32be06a6f89b2893be06a23b2463
I20260923 17:28:00.157243 8398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c: Bootstrap starting.
I20260923 17:28:00.161648 8398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:00.162954 8398 log.cc:824] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:00.165985 8398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c: No bootstrap required, opened a new log
I20260923 17:28:00.176501 8398 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f485319943b42309a63d15c20f0397c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38833 } }
I20260923 17:28:00.176885 8398 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:00.176981 8398 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f485319943b42309a63d15c20f0397c, State: Initialized, Role: FOLLOWER
I20260923 17:28:00.178589 8398 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f485319943b42309a63d15c20f0397c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38833 } }
I20260923 17:28:00.179106 8403 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:00.179569 8403 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f485319943b42309a63d15c20f0397c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38833 } }
I20260923 17:28:00.180474 8403 leader_election.cc:302] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f485319943b42309a63d15c20f0397c; no voters:
I20260923 17:28:00.178939 8398 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:00.181269 8398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:00.181437 8398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:00.181578 8403 leader_election.cc:288] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260923 17:28:00.183862 8398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f485319943b42309a63d15c20f0397c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38833 } }
I20260923 17:28:00.184098 8398 leader_election.cc:302] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f485319943b42309a63d15c20f0397c; no voters:
I20260923 17:28:00.184154 8404 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260923 17:28:00.184336 8398 leader_election.cc:288] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:00.184645 8403 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:00.185204 8403 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [term 1 LEADER]: Becoming Leader. State: Replica: 6f485319943b42309a63d15c20f0397c, State: Running, Role: LEADER
I20260923 17:28:00.185830 8403 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f485319943b42309a63d15c20f0397c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38833 } }
I20260923 17:28:00.187300 8398 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [sys.catalog]: configured and running, proceeding with master startup.
Sep 23 17:28:00 dist-test-slave-p7jt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184479, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:00.187832 8406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f485319943b42309a63d15c20f0397c. Latest consensus state: current_term: 1 leader_uuid: "6f485319943b42309a63d15c20f0397c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f485319943b42309a63d15c20f0397c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38833 } } }
I20260923 17:28:00.188304 8406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:00.187840 8404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f485319943b42309a63d15c20f0397c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f485319943b42309a63d15c20f0397c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38833 } } }
I20260923 17:28:00.188674 8404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:00.189177 8408 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:00.202090 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:00.158371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.8.29.254:38833)
0923 17:28:00.159263 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:00.159280 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:00.160547 (+ 1267us) server_negotiation.cc:407] Connection header received
0923 17:28:00.172496 (+ 11949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:00.172524 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:00.173050 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:00.173503 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:00.177851 (+ 4348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.179688 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.181575 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.182409 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.182605 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:00.189687 (+ 7082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:00.189700 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:00.189724 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:00.189784 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:00.194273 (+ 4489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.195650 (+ 1377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.195661 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.195671 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.195985 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.196929 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.196935 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.196940 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.198053 (+ 1113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:00.198383 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:00.199936 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0923 17:28:00.200482 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":649,"thread_start_us":396,"threads_started":1}
I20260923 17:28:00.205875 8408 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:00.215713 8408 catalog_manager.cc:1347] Generated new cluster ID: 14eed90b780e4e91ba2977514d102b48
I20260923 17:28:00.215865 8408 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:00.226678 8408 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:00.228003 8408 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:00.236886 8408 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f485319943b42309a63d15c20f0397c: Generated new TSK 0
I20260923 17:28:00.237849 8408 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:28:00.285105 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38833
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:00.644212 8425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:00.644522 8425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:00.644620 8425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:00.652400 8425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:00.652544 8425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:00.652684 8425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:00.663235 8425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:00.664546 8425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:00.665803 8425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:00.678706 8431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:00.678728 8433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:00.678761 8430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:00.679337 8425 server_base.cc:1034] running on GCE node
I20260923 17:28:00.679911 8425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:00.680882 8425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:00.682109 8425 hybrid_clock.cc:648] HybridClock initialized: now 1790184480682042 us; error 58 us; skew 500 ppm
I20260923 17:28:00.682488 8425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:00 dist-test-slave-p7jt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:00.686633 8425 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:28:00.689221 8425 webserver.cc:460] Webserver started at http://127.8.29.193:33595/ using document root <none> and password file <none>
I20260923 17:28:00.690116 8425 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:00.690235 8425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:00.690577 8425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:00.694541 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "6bd5c160696f49c48a1eff9a0015f4be"
format_stamp: "Formatted at 2026-09-23 17:28:00 on dist-test-slave-p7jt"
server_key: "f3938f1d2cb4f4d13d1efa777668325a"
server_key_iv: "e4fbf4511fd13c94d819eef19badbcd4"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.695595 8425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "6bd5c160696f49c48a1eff9a0015f4be"
format_stamp: "Formatted at 2026-09-23 17:28:00 on dist-test-slave-p7jt"
server_key: "f3938f1d2cb4f4d13d1efa777668325a"
server_key_iv: "e4fbf4511fd13c94d819eef19badbcd4"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.702781 8425 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260923 17:28:00.706894 8440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.708443 8425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:00.708611 8425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "6bd5c160696f49c48a1eff9a0015f4be"
format_stamp: "Formatted at 2026-09-23 17:28:00 on dist-test-slave-p7jt"
server_key: "f3938f1d2cb4f4d13d1efa777668325a"
server_key_iv: "e4fbf4511fd13c94d819eef19badbcd4"
server_key_version: "encryptionkey@0"
I20260923 17:28:00.708815 8425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:00.725433 8425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:00.729936 8425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:00.730221 8425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:00.731330 8425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:00.733053 8425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:00.733129 8425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.733242 8425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:00.733299 8425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:00.768630 8425 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:44453
I20260923 17:28:00.768661 8552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:44453 every 8 connection(s)
I20260923 17:28:00.769997 8425 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:00.772289 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 8425
I20260923 17:28:00.772495 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:00.773277 8311 external_mini_cluster.cc:1428] Setting key d9b9a537069edefb1734d05d5c421870
Sep 23 17:28:00 dist-test-slave-p7jt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184480, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
I20260923 17:28:00.794672 8555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:00.772804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:47691 (local address 127.8.29.254:38833)
0923 17:28:00.773325 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:00.773331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:00.774449 (+ 1118us) server_negotiation.cc:407] Connection header received
0923 17:28:00.776912 (+ 2463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:00.776916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:00.776984 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:00.777084 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:00.779250 (+ 2166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.779974 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.781853 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.782314 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.782424 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:00.786710 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:00.786718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:00.786723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:00.786769 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:00.791220 (+ 4451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.792122 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.792127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.792129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.792255 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:00.792711 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:00.792715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:00.792717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:00.793059 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:00.793192 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:00.794015 (+ 823us) server_negotiation.cc:299] Negotiation successful
0923 17:28:00.794198 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":270,"threads_started":1}
I20260923 17:28:00.795967 8556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:00.773421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38833 (local address 127.8.29.193:47691)
0923 17:28:00.774263 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:00.774317 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:00.776579 (+ 2262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:00.777388 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:00.777402 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:00.778195 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:00.779040 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:00.779080 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.780152 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.780156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:00.781694 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:00.781710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:00.782581 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:00.782589 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:00.782877 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:00.783626 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:00.783655 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:00.786382 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:00.791401 (+ 5019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:00.791409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:00.791425 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:00.791958 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:00.792369 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:00.792374 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:00.792376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:00.792577 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:00.793186 (+ 609us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:00.793194 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:00.793571 (+ 377us) client_negotiation.cc:769] Sending connection context
0923 17:28:00.793829 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 17:28:00.794385 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":326,"threads_started":1}
I20260923 17:28:00.797760 8553 heartbeater.cc:347] Connected to a master server at 127.8.29.254:38833
I20260923 17:28:00.798084 8553 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:00.798921 8553 heartbeater.cc:511] Master 127.8.29.254:38833 requested a full tablet report, sending...
I20260923 17:28:00.801185 8363 ts_manager.cc:194] Registered new tserver with Master: 6bd5c160696f49c48a1eff9a0015f4be (127.8.29.193:44453)
I20260923 17:28:00.803108 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:47691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:28:00.819574 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38833
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:01.152070 8559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:01.152470 8559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:01.152660 8559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:01.162395 8559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:01.162581 8559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:01.162729 8559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:01.173383 8559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:01.174731 8559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:01.176030 8559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:01.189572 8564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:01.189630 8567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:01.189597 8565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:01.190603 8559 server_base.cc:1034] running on GCE node
I20260923 17:28:01.191262 8559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:01.192296 8559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:01.193521 8559 hybrid_clock.cc:648] HybridClock initialized: now 1790184481193468 us; error 38 us; skew 500 ppm
I20260923 17:28:01.193866 8559 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:01 dist-test-slave-p7jt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:01.197192 8559 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:28:01.198726 8559 webserver.cc:460] Webserver started at http://127.8.29.194:44525/ using document root <none> and password file <none>
I20260923 17:28:01.199430 8559 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:01.199538 8559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:01.199855 8559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:01.203282 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "723d89242ba541c8a25b00d2ba701f67"
format_stamp: "Formatted at 2026-09-23 17:28:01 on dist-test-slave-p7jt"
server_key: "6cdac008a50e895bf2cd16d51ec1cb5f"
server_key_iv: "a0e768152e273b67bac1bd40bc7c20bd"
server_key_version: "encryptionkey@0"
I20260923 17:28:01.204180 8559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "723d89242ba541c8a25b00d2ba701f67"
format_stamp: "Formatted at 2026-09-23 17:28:01 on dist-test-slave-p7jt"
server_key: "6cdac008a50e895bf2cd16d51ec1cb5f"
server_key_iv: "a0e768152e273b67bac1bd40bc7c20bd"
server_key_version: "encryptionkey@0"
I20260923 17:28:01.209774 8559 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 17:28:01.214169 8574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:01.215819 8559 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:28:01.216022 8559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "723d89242ba541c8a25b00d2ba701f67"
format_stamp: "Formatted at 2026-09-23 17:28:01 on dist-test-slave-p7jt"
server_key: "6cdac008a50e895bf2cd16d51ec1cb5f"
server_key_iv: "a0e768152e273b67bac1bd40bc7c20bd"
server_key_version: "encryptionkey@0"
I20260923 17:28:01.216252 8559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:01.256714 8559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:01.260947 8559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:01.261245 8559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:01.262279 8559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:01.263724 8559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:01.263804 8559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:01.263934 8559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:01.263990 8559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:01.295115 8559 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:40201
I20260923 17:28:01.295143 8686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:40201 every 8 connection(s)
I20260923 17:28:01.296530 8559 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:01.306809 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 8559
I20260923 17:28:01.306986 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:01.307400 8311 external_mini_cluster.cc:1428] Setting key 46f0ea228f24a371d8e73cff34ebe175
Sep 23 17:28:01 dist-test-slave-p7jt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:01.317062 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.299279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:34489 (local address 127.8.29.254:38833)
0923 17:28:01.299730 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.299734 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.300704 (+ 970us) server_negotiation.cc:407] Connection header received
0923 17:28:01.302554 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.302559 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.302643 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.302737 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:01.304297 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.305021 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.306614 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.307095 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.307212 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.310990 (+ 3778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:01.310996 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:01.310999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:01.311038 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:01.313713 (+ 2675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.314581 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.314586 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.314590 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.314729 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.315176 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.315179 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.315181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.315457 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:01.315607 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.316404 (+ 797us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.316649 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":229,"threads_started":1}
I20260923 17:28:01.318136 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.299815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38833 (local address 127.8.29.194:34489)
0923 17:28:01.300540 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:01.300582 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:01.302234 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:01.302992 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:01.303001 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:01.303545 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:01.304123 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.304143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.305175 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.305179 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.306426 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.306439 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.307324 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.307331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.307594 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.308258 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:01.308281 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:01.310710 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:01.313891 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:01.313902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:01.313938 (+ 36us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:01.314443 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:01.314842 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:01.314847 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:01.314852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:01.315056 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:01.315589 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:01.315598 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:01.315984 (+ 386us) client_negotiation.cc:769] Sending connection context
0923 17:28:01.316283 (+ 299us) client_negotiation.cc:240] Negotiation successful
0923 17:28:01.316915 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":263,"threads_started":1}
I20260923 17:28:01.319674 8687 heartbeater.cc:347] Connected to a master server at 127.8.29.254:38833
I20260923 17:28:01.320093 8687 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:01.321087 8687 heartbeater.cc:511] Master 127.8.29.254:38833 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
I20260923 17:28:01.322844 8363 ts_manager.cc:194] Registered new tserver with Master: 723d89242ba541c8a25b00d2ba701f67 (127.8.29.194:40201)
I20260923 17:28:01.323704 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:34489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:28:01.346766 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38833
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:01.690716 8693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:01.691023 8693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:01.691216 8693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:01.698644 8693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:01.698897 8693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:01.699092 8693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:01.709183 8693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38833
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:01.710624 8693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:01.712074 8693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:01.726819 8698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:01.726850 8699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:01.726997 8701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:01.727696 8693 server_base.cc:1034] running on GCE node
I20260923 17:28:01.728324 8693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:01.729337 8693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:01.730584 8693 hybrid_clock.cc:648] HybridClock initialized: now 1790184481730523 us; error 46 us; skew 500 ppm
I20260923 17:28:01.730926 8693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:01 dist-test-slave-p7jt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:01.733964 8693 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:28:01.735450 8693 webserver.cc:460] Webserver started at http://127.8.29.195:45931/ using document root <none> and password file <none>
I20260923 17:28:01.736142 8693 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:01.736258 8693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:01.736557 8693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:01.739574 8693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "7f82c0efb1834151b0bcf5c27587bf32"
format_stamp: "Formatted at 2026-09-23 17:28:01 on dist-test-slave-p7jt"
server_key: "f16189a23d4c54dd6f0fe589f31c147e"
server_key_iv: "1d5c8e364ebdec36ba40886187248dc0"
server_key_version: "encryptionkey@0"
I20260923 17:28:01.740343 8693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "7f82c0efb1834151b0bcf5c27587bf32"
format_stamp: "Formatted at 2026-09-23 17:28:01 on dist-test-slave-p7jt"
server_key: "f16189a23d4c54dd6f0fe589f31c147e"
server_key_iv: "1d5c8e364ebdec36ba40886187248dc0"
server_key_version: "encryptionkey@0"
I20260923 17:28:01.745795 8693 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 17:28:01.749827 8708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:01.751391 8693 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:01.751576 8693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "7f82c0efb1834151b0bcf5c27587bf32"
format_stamp: "Formatted at 2026-09-23 17:28:01 on dist-test-slave-p7jt"
server_key: "f16189a23d4c54dd6f0fe589f31c147e"
server_key_iv: "1d5c8e364ebdec36ba40886187248dc0"
server_key_version: "encryptionkey@0"
I20260923 17:28:01.751808 8693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:01.768209 8693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:01.773255 8693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:01.773542 8693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:01.774607 8693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:01.776050 8693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:01.776129 8693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:01.776250 8693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:01.776315 8693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:01.806957 8553 heartbeater.cc:503] Master 127.8.29.254:38833 was elected leader, sending a full tablet report...
I20260923 17:28:01.809832 8693 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:36423
I20260923 17:28:01.809883 8820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:36423 every 8 connection(s)
I20260923 17:28:01.811179 8693 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:01.813019 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 8693
I20260923 17:28:01.813243 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:01.813771 8311 external_mini_cluster.cc:1428] Setting key db4ba38817667ef74525cfa3d9363e54
Sep 23 17:28:01 dist-test-slave-p7jt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:01.832780 8689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.814115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:45237 (local address 127.8.29.254:38833)
0923 17:28:01.814364 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.814368 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.815560 (+ 1192us) server_negotiation.cc:407] Connection header received
0923 17:28:01.817972 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.817978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.818079 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.818172 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:01.819981 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.820801 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.822574 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.822976 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.823099 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.826802 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:01.826808 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:01.826812 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:01.826850 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:01.829676 (+ 2826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.830664 (+ 988us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.830670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.830673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.830819 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.831241 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.831244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.831247 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.831574 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:01.831691 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.832333 (+ 642us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.832528 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260923 17:28:01.833767 8823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.814492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38833 (local address 127.8.29.195:45237)
0923 17:28:01.815395 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:01.815449 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:01.817655 (+ 2206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:01.818435 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:01.818447 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:01.819126 (+ 679us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:01.819778 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.819799 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.820955 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.820959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.822288 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:01.822301 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.823212 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.823217 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:01.823461 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.824159 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:01.824182 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:01.826549 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:01.829831 (+ 3282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:01.829838 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:01.829855 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:01.830490 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:01.830910 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:01.830913 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:01.830915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:01.831127 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:01.831668 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:01.831673 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:01.832014 (+ 341us) client_negotiation.cc:769] Sending connection context
0923 17:28:01.832240 (+ 226us) client_negotiation.cc:240] Negotiation successful
0923 17:28:01.832673 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":346,"threads_started":1}
I20260923 17:28:01.835271 8821 heartbeater.cc:347] Connected to a master server at 127.8.29.254:38833
I20260923 17:28:01.835578 8821 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:01.836308 8821 heartbeater.cc:511] Master 127.8.29.254:38833 requested a full tablet report, sending...
I20260923 17:28:01.837779 8363 ts_manager.cc:194] Registered new tserver with Master: 7f82c0efb1834151b0bcf5c27587bf32 (127.8.29.195:36423)
I20260923 17:28:01.838665 8363 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:45237
I20260923 17:28:01.843309 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:28:01 dist-test-slave-p7jt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:28:01 dist-test-slave-p7jt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:01.886768 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.870932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32836 (local address 127.8.29.193:44453)
0923 17:28:01.871619 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.871629 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.871662 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:28:01.871797 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.871805 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.872042 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.872187 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:01.873816 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.874923 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.876187 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.876782 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.876895 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.880829 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:01.880843 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:01.880860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:01.880901 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:01.883875 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.884588 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.884597 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.884599 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.884736 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.885143 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.885147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.885149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.885520 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:01.885818 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.886118 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.886387 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":335,"threads_started":1}
W20260923 17:28:01.888085 8468 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32836
Sep 23 17:28:01 dist-test-slave-p7jt krb5kdc[8318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:28:01 dist-test-slave-p7jt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:01.932832 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.917162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.8.29.193:44453)
0923 17:28:01.917315 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.917319 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.917730 (+ 411us) server_negotiation.cc:407] Connection header received
0923 17:28:01.917910 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.917914 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.917980 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.918076 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:01.919688 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.920554 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.921918 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.922399 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.922547 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.926863 (+ 4316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:01.926868 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:01.926872 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:01.926917 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:01.929590 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.930549 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.930573 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.930577 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.930737 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.931399 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.931404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.931406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.931718 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:01.932016 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.932370 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.932580 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:28:01.934564 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 8425
I20260923 17:28:01.948011 8828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:01.935588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32850 (local address 127.8.29.193:44453)
0923 17:28:01.935736 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:01.935740 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:01.936022 (+ 282us) server_negotiation.cc:407] Connection header received
0923 17:28:01.936156 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:01.936159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:01.936218 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:01.936295 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:01.937448 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.938184 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.939504 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:01.939968 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:01.940077 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:01.942580 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:01.942586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:01.942589 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:01.942626 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:01.945271 (+ 2645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.946078 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.946083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.946085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.946239 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:01.946764 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:01.946769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:01.946771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:01.947060 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:01.947286 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:01.947559 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 17:28:01.947742 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 17:28:01.948804 8548 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:02.032719 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 8425
I20260923 17:28:02.046103 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 8559
Sep 23 17:28:02 dist-test-slave-p7jt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:28:02.062299 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:02.046995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47152 (local address 127.8.29.194:40201)
0923 17:28:02.047505 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:02.047512 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:02.047538 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:28:02.047598 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:02.047605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:02.047809 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:02.047983 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:02.049106 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:02.050142 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:02.051470 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:02.052050 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:02.052156 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:02.056191 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:02.056205 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:02.056222 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:02.056278 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:02.059219 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:02.060070 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:02.060079 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:02.060081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:02.060228 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:02.060676 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:02.060680 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:02.060682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:02.061109 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:02.061421 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:02.061728 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 17:28:02.061941 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":251,"threads_started":1}
I20260923 17:28:02.063591 8682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:02.145991 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 8559
I20260923 17:28:02.166591 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 8693
Sep 23 17:28:02 dist-test-slave-p7jt krb5kdc[8318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184481, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:28:02.183480 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:02.167693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55888 (local address 127.8.29.195:36423)
0923 17:28:02.168294 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:02.168302 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:02.168332 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:28:02.168394 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:02.168400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:02.168629 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:02.168784 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:02.170125 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:02.171243 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:02.172641 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:02.173225 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:02.173345 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:02.177588 (+ 4243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:02.177603 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:02.177619 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:02.177659 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:02.180603 (+ 2944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:02.181349 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:02.181357 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:02.181360 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:02.181495 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:02.181863 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:02.181867 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:02.181869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:02.182227 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:02.182573 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:02.182863 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 17:28:02.183135 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":292,"threads_started":1}
I20260923 17:28:02.184825 8816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:02.267186 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 8693
I20260923 17:28:02.282143 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 8331
I20260923 17:28:02.283545 8393 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:02.386054 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 8331
2026-09-23T17:28:02Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3874 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:02 dist-test-slave-p7jt krb5kdc[8842](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:02 dist-test-slave-p7jt krb5kdc[8842](info): set up 2 sockets
Sep 23 17:28:02 dist-test-slave-p7jt krb5kdc[8842](info): commencing operation
krb5kdc: starting...
W20260923 17:28:03.398307 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.974s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:03 dist-test-slave-p7jt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:28:03.570811 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39547
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:39547
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:03.886343 8855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:03.886713 8855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:03.886842 8855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:03.894179 8855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:03.894320 8855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:03.894388 8855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:03.894481 8855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:03.894544 8855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:03.904564 8855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:39547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:03.905964 8855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:03.907346 8855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:03.918774 8860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:03.918872 8861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:03.918875 8863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:03.919399 8855 server_base.cc:1034] running on GCE node
I20260923 17:28:03.920434 8855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:03.921907 8855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:03.923141 8855 hybrid_clock.cc:648] HybridClock initialized: now 1790184483923100 us; error 61 us; skew 500 ppm
I20260923 17:28:03.923491 8855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:03 dist-test-slave-p7jt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184483, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:03.931252 8855 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:03.932946 8855 webserver.cc:460] Webserver started at http://127.8.29.254:36845/ using document root <none> and password file <none>
I20260923 17:28:03.933665 8855 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:03.933785 8855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:03.934085 8855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:03.937129 8855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "0b4ebda4a8ea437e92a161ce85ac3c72"
format_stamp: "Formatted at 2026-09-23 17:28:03 on dist-test-slave-p7jt"
server_key: "8b0263d0bd8cad11bc93c7d76f4e5208"
server_key_iv: "27eb50f65d740db5e11acbb92d739e28"
server_key_version: "encryptionkey@0"
I20260923 17:28:03.937873 8855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "0b4ebda4a8ea437e92a161ce85ac3c72"
format_stamp: "Formatted at 2026-09-23 17:28:03 on dist-test-slave-p7jt"
server_key: "8b0263d0bd8cad11bc93c7d76f4e5208"
server_key_iv: "27eb50f65d740db5e11acbb92d739e28"
server_key_version: "encryptionkey@0"
I20260923 17:28:03.943241 8855 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 17:28:03.946781 8870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:03.948282 8855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:28:03.948460 8855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "0b4ebda4a8ea437e92a161ce85ac3c72"
format_stamp: "Formatted at 2026-09-23 17:28:03 on dist-test-slave-p7jt"
server_key: "8b0263d0bd8cad11bc93c7d76f4e5208"
server_key_iv: "27eb50f65d740db5e11acbb92d739e28"
server_key_version: "encryptionkey@0"
I20260923 17:28:03.948654 8855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:03.976817 8855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:03.981426 8855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:03.981736 8855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:03.999336 8855 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:39547
I20260923 17:28:03.999357 8921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:39547 every 8 connection(s)
I20260923 17:28:04.000734 8855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:04.004596 8922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:04.007714 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 8855
I20260923 17:28:04.007947 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:04.008721 8311 external_mini_cluster.cc:1428] Setting key a12849fa97a6873b96b9edfd45647822
I20260923 17:28:04.016937 8922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72: Bootstrap starting.
I20260923 17:28:04.020105 8922 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:04.021138 8922 log.cc:824] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:28:04 dist-test-slave-p7jt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:04.024057 8922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72: No bootstrap required, opened a new log
I20260923 17:28:04.030834 8925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:04.010985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52034 (local address 127.8.29.254:39547)
0923 17:28:04.011744 (+ 759us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:04.011757 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:04.011798 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:28:04.013460 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:04.013484 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:04.013901 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:04.014304 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:04.015817 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:04.017180 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:04.018435 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:04.019204 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:04.019382 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:04.023292 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:04.023300 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:04.023317 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:04.023358 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:04.026511 (+ 3153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:04.027240 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:04.027251 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:04.027258 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:04.027438 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:04.027816 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:04.027820 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:04.027822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:04.028417 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:04.028711 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:04.029276 (+ 565us) server_negotiation.cc:299] Negotiation successful
0923 17:28:04.029696 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":322,"threads_started":1}
I20260923 17:28:04.032240 8922 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ebda4a8ea437e92a161ce85ac3c72" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39547 } }
I20260923 17:28:04.032670 8922 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:04.032769 8922 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b4ebda4a8ea437e92a161ce85ac3c72, State: Initialized, Role: FOLLOWER
I20260923 17:28:04.033326 8922 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ebda4a8ea437e92a161ce85ac3c72" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39547 } }
I20260923 17:28:04.033562 8922 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:04.033685 8922 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:04.033849 8922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:04.036264 8922 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ebda4a8ea437e92a161ce85ac3c72" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39547 } }
I20260923 17:28:04.036824 8922 leader_election.cc:302] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b4ebda4a8ea437e92a161ce85ac3c72; no voters:
I20260923 17:28:04.037381 8922 leader_election.cc:288] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:04.037515 8927 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:04.038143 8927 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [term 1 LEADER]: Becoming Leader. State: Replica: 0b4ebda4a8ea437e92a161ce85ac3c72, State: Running, Role: LEADER
I20260923 17:28:04.038750 8927 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ebda4a8ea437e92a161ce85ac3c72" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39547 } }
I20260923 17:28:04.039410 8922 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:04.041466 8928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b4ebda4a8ea437e92a161ce85ac3c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ebda4a8ea437e92a161ce85ac3c72" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39547 } } }
I20260923 17:28:04.041504 8929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b4ebda4a8ea437e92a161ce85ac3c72. Latest consensus state: current_term: 1 leader_uuid: "0b4ebda4a8ea437e92a161ce85ac3c72" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b4ebda4a8ea437e92a161ce85ac3c72" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39547 } } }
I20260923 17:28:04.041922 8928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:04.041930 8929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:04.042833 8934 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:04.048832 8934 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:04.058010 8934 catalog_manager.cc:1347] Generated new cluster ID: 718d31a318ad4cafb0e296f4d0124f34
I20260923 17:28:04.058130 8934 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:04.072441 8934 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:04.073941 8934 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:04.093135 8934 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b4ebda4a8ea437e92a161ce85ac3c72: Generated new TSK 0
I20260923 17:28:04.094234 8934 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:28:04.146435 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39547
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:04.463001 8948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:04.463291 8948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:04.463384 8948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:04.470533 8948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:04.470645 8948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:04.470760 8948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:04.480386 8948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39547
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:04.481655 8948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:04.482942 8948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:04.495489 8954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:04.495563 8956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:04.495501 8953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:04.496598 8948 server_base.cc:1034] running on GCE node
I20260923 17:28:04.497222 8948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:04.498142 8948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:04.499336 8948 hybrid_clock.cc:648] HybridClock initialized: now 1790184484499286 us; error 34 us; skew 500 ppm
I20260923 17:28:04.499637 8948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:04 dist-test-slave-p7jt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:04.502712 8948 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:28:04.504204 8948 webserver.cc:460] Webserver started at http://127.8.29.193:44237/ using document root <none> and password file <none>
I20260923 17:28:04.504874 8948 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:04.504997 8948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:04.505278 8948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:04.508234 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "6becb83013584aec90bdee9ac2b3beaf"
format_stamp: "Formatted at 2026-09-23 17:28:04 on dist-test-slave-p7jt"
server_key: "eb808d8ba9e2827315314d7c5c740ed2"
server_key_iv: "44763f8f5776e0451233713b30c328bd"
server_key_version: "encryptionkey@0"
I20260923 17:28:04.509034 8948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "6becb83013584aec90bdee9ac2b3beaf"
format_stamp: "Formatted at 2026-09-23 17:28:04 on dist-test-slave-p7jt"
server_key: "eb808d8ba9e2827315314d7c5c740ed2"
server_key_iv: "44763f8f5776e0451233713b30c328bd"
server_key_version: "encryptionkey@0"
I20260923 17:28:04.514318 8948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:28:04.517818 8963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:04.519321 8948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:04.519479 8948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "6becb83013584aec90bdee9ac2b3beaf"
format_stamp: "Formatted at 2026-09-23 17:28:04 on dist-test-slave-p7jt"
server_key: "eb808d8ba9e2827315314d7c5c740ed2"
server_key_iv: "44763f8f5776e0451233713b30c328bd"
server_key_version: "encryptionkey@0"
I20260923 17:28:04.519695 8948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:04.554617 8948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:04.558920 8948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:04.559191 8948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:04.560168 8948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:04.561537 8948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:04.561615 8948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:04.561740 8948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:04.561795 8948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:04.592098 8948 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:38895
I20260923 17:28:04.592196 9075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:38895 every 8 connection(s)
I20260923 17:28:04.593514 8948 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:04.602918 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 8948
I20260923 17:28:04.603118 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:04.603708 8311 external_mini_cluster.cc:1428] Setting key c1aaa7a183c8a8593f1b6756765e24f8
Sep 23 17:28:04 dist-test-slave-p7jt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:04.615082 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:04.596530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:51669 (local address 127.8.29.254:39547)
0923 17:28:04.596992 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:04.596998 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:04.598103 (+ 1105us) server_negotiation.cc:407] Connection header received
0923 17:28:04.599989 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:04.599995 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:04.600098 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:04.600204 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:04.601761 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:04.602585 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:04.604314 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:04.604785 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:04.604913 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:04.608965 (+ 4052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:04.608971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:04.608974 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:04.609010 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:04.612030 (+ 3020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:04.612800 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:04.612805 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:04.612808 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:04.612947 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:04.613320 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:04.613325 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:04.613328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:04.613656 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:04.613791 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:04.614500 (+ 709us) server_negotiation.cc:299] Negotiation successful
0923 17:28:04.614720 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":216,"threads_started":1}
I20260923 17:28:04.616026 9079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:04.597232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39547 (local address 127.8.29.193:51669)
0923 17:28:04.597923 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:04.597961 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:04.599710 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:04.600474 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:04.600483 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:04.601006 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:04.601596 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:04.601613 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:04.602742 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:04.602746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:04.604159 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:04.604175 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:04.605025 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:04.605030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:04.605233 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:04.605930 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:04.605969 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:04.608709 (+ 2740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:04.612199 (+ 3490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:04.612206 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:04.612220 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:04.612685 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:04.613043 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:04.613046 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:04.613048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:04.613225 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:04.613759 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:04.613765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:04.614120 (+ 355us) client_negotiation.cc:769] Sending connection context
0923 17:28:04.614349 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 17:28:04.614798 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":281,"threads_started":1}
I20260923 17:28:04.617697 9076 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39547
I20260923 17:28:04.617987 9076 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:04.618722 9076 heartbeater.cc:511] Master 127.8.29.254:39547 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:28:04.620839 8887 ts_manager.cc:194] Registered new tserver with Master: 6becb83013584aec90bdee9ac2b3beaf (127.8.29.193:38895)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
I20260923 17:28:04.622644 8887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:51669
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:28:04.645282 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39547
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:04.959415 9082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:04.959721 9082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:04.959858 9082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:04.967231 9082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:04.967360 9082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:04.967510 9082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:04.977666 9082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39547
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:04.979075 9082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:04.980548 9082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:04.994069 9088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:04.994102 9087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:04.994036 9090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:04.995122 9082 server_base.cc:1034] running on GCE node
I20260923 17:28:04.995949 9082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:04.997144 9082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:04.998413 9082 hybrid_clock.cc:648] HybridClock initialized: now 1790184484998343 us; error 57 us; skew 500 ppm
I20260923 17:28:04.998838 9082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:04 dist-test-slave-p7jt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:05.002394 9082 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:28:05.004189 9082 webserver.cc:460] Webserver started at http://127.8.29.194:36681/ using document root <none> and password file <none>
I20260923 17:28:05.004992 9082 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:05.005127 9082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:05.005455 9082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:05.008802 9082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "aa65ecd800dc408f9c614de091269864"
format_stamp: "Formatted at 2026-09-23 17:28:05 on dist-test-slave-p7jt"
server_key: "60c49215022bf0ed3d8c6b289e53ce31"
server_key_iv: "d22c53219a76334934f8bd47c161a625"
server_key_version: "encryptionkey@0"
I20260923 17:28:05.009678 9082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "aa65ecd800dc408f9c614de091269864"
format_stamp: "Formatted at 2026-09-23 17:28:05 on dist-test-slave-p7jt"
server_key: "60c49215022bf0ed3d8c6b289e53ce31"
server_key_iv: "d22c53219a76334934f8bd47c161a625"
server_key_version: "encryptionkey@0"
I20260923 17:28:05.015201 9082 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:28:05.019333 9097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:05.020955 9082 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20260923 17:28:05.021147 9082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "aa65ecd800dc408f9c614de091269864"
format_stamp: "Formatted at 2026-09-23 17:28:05 on dist-test-slave-p7jt"
server_key: "60c49215022bf0ed3d8c6b289e53ce31"
server_key_iv: "d22c53219a76334934f8bd47c161a625"
server_key_version: "encryptionkey@0"
I20260923 17:28:05.021348 9082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:05.035997 9082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:05.040345 9082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:05.040619 9082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:05.041600 9082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:05.042990 9082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:05.043067 9082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:05.043181 9082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:05.043236 9082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:05.073740 9082 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:36329
I20260923 17:28:05.073789 9209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:36329 every 8 connection(s)
I20260923 17:28:05.075205 9082 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:05.081489 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 9082
I20260923 17:28:05.081696 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:05.082212 8311 external_mini_cluster.cc:1428] Setting key 4aeeb83f2801dac717a64102b479e41b
Sep 23 17:28:05 dist-test-slave-p7jt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184484, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:05.097066 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.077894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:48903 (local address 127.8.29.254:39547)
0923 17:28:05.078091 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.078096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.079625 (+ 1529us) server_negotiation.cc:407] Connection header received
0923 17:28:05.081599 (+ 1974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.081606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.081697 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.081807 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:05.083667 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.084631 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.086975 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.087430 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.087533 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.091389 (+ 3856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:05.091394 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:05.091397 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:05.091431 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:05.093887 (+ 2456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.094746 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.094751 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.094753 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.094886 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.095325 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.095329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.095332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.095622 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:05.095771 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.096482 (+ 711us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.096724 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:28:05.098105 9212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.078272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39547 (local address 127.8.29.194:48903)
0923 17:28:05.079423 (+ 1151us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:05.079481 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:05.081233 (+ 1752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:05.082080 (+ 847us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:05.082093 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:05.082886 (+ 793us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:05.083496 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.083515 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.084791 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.084797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.086823 (+ 2026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.086839 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.087658 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.087665 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.087941 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.088634 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:05.088662 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:05.091134 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:05.094093 (+ 2959us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:05.094104 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:05.094122 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:05.094616 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:05.094999 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:05.095004 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:05.095008 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:05.095207 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:05.095747 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:05.095755 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:05.096116 (+ 361us) client_negotiation.cc:769] Sending connection context
0923 17:28:05.096349 (+ 233us) client_negotiation.cc:240] Negotiation successful
0923 17:28:05.096941 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":451,"threads_started":1}
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
I20260923 17:28:05.099608 9210 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39547
I20260923 17:28:05.099910 9210 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:05.100729 9210 heartbeater.cc:511] Master 127.8.29.254:39547 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
I20260923 17:28:05.102416 8887 ts_manager.cc:194] Registered new tserver with Master: aa65ecd800dc408f9c614de091269864 (127.8.29.194:36329)
I20260923 17:28:05.103245 8887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:48903
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:28:05.122249 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39547
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:05.436026 9215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:05.436324 9215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:05.436451 9215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:05.443827 9215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:05.443949 9215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:05.444101 9215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:05.454193 9215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39547
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:05.455531 9215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:05.456852 9215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:05.469722 9220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:05.469692 9221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:05.469653 9223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:05.470546 9215 server_base.cc:1034] running on GCE node
I20260923 17:28:05.471131 9215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:05.472088 9215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:05.473292 9215 hybrid_clock.cc:648] HybridClock initialized: now 1790184485473237 us; error 46 us; skew 500 ppm
I20260923 17:28:05.473614 9215 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:05 dist-test-slave-p7jt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184485, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:05.476691 9215 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:28:05.478214 9215 webserver.cc:460] Webserver started at http://127.8.29.195:39749/ using document root <none> and password file <none>
I20260923 17:28:05.478950 9215 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:05.479077 9215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:05.479382 9215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:05.482360 9215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "f01bc01166ee4993aee95fd868f9039a"
format_stamp: "Formatted at 2026-09-23 17:28:05 on dist-test-slave-p7jt"
server_key: "6c48474eeaef98ba713faa1baf9f3e0e"
server_key_iv: "9c691c2c1af02f668b4a596f57859ec5"
server_key_version: "encryptionkey@0"
I20260923 17:28:05.483230 9215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "f01bc01166ee4993aee95fd868f9039a"
format_stamp: "Formatted at 2026-09-23 17:28:05 on dist-test-slave-p7jt"
server_key: "6c48474eeaef98ba713faa1baf9f3e0e"
server_key_iv: "9c691c2c1af02f668b4a596f57859ec5"
server_key_version: "encryptionkey@0"
I20260923 17:28:05.488693 9215 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:05.492479 9230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:05.493908 9215 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:05.494102 9215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "f01bc01166ee4993aee95fd868f9039a"
format_stamp: "Formatted at 2026-09-23 17:28:05 on dist-test-slave-p7jt"
server_key: "6c48474eeaef98ba713faa1baf9f3e0e"
server_key_iv: "9c691c2c1af02f668b4a596f57859ec5"
server_key_version: "encryptionkey@0"
I20260923 17:28:05.494288 9215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:05.518738 9215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:05.523242 9215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:05.523484 9215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:05.524466 9215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:05.525959 9215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:05.526039 9215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:05.526127 9215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:05.526183 9215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:05.556674 9215 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:43069
I20260923 17:28:05.556728 9342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:43069 every 8 connection(s)
I20260923 17:28:05.557902 9215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:05.559077 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 9215
I20260923 17:28:05.559267 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:05.559816 8311 external_mini_cluster.cc:1428] Setting key 46626d64c0c5b2905b15803185b51424
Sep 23 17:28:05 dist-test-slave-p7jt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184485, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:05.578703 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.560772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:35245 (local address 127.8.29.254:39547)
0923 17:28:05.560937 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.560941 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.562222 (+ 1281us) server_negotiation.cc:407] Connection header received
0923 17:28:05.564254 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.564259 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.564348 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.564430 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:05.566054 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.566858 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.568395 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.568792 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.568909 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.572767 (+ 3858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:05.572773 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:05.572777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:05.572820 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:05.575631 (+ 2811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.576365 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.576369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.576372 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.576507 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.576950 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.576954 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.576956 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.577242 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:05.577372 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.578109 (+ 737us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.578332 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:28:05.579741 9345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.561326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39547 (local address 127.8.29.195:35245)
0923 17:28:05.562073 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:05.562116 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:05.563990 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:05.564697 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:05.564706 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:05.565229 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:05.565867 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.565886 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.566999 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.567005 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.568254 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.568267 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.569060 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.569067 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.569295 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.569964 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:05.569994 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:05.572367 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:05.575779 (+ 3412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:05.575790 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:05.575809 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:05.576248 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:05.576649 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:05.576652 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:05.576655 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:05.576843 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:05.577370 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:05.577380 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:05.577744 (+ 364us) client_negotiation.cc:769] Sending connection context
0923 17:28:05.577997 (+ 253us) client_negotiation.cc:240] Negotiation successful
0923 17:28:05.578588 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":351,"threads_started":1}
I20260923 17:28:05.581226 9343 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39547
I20260923 17:28:05.581586 9343 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:05.582305 9343 heartbeater.cc:511] Master 127.8.29.254:39547 requested a full tablet report, sending...
I20260923 17:28:05.583825 8887 ts_manager.cc:194] Registered new tserver with Master: f01bc01166ee4993aee95fd868f9039a (127.8.29.195:43069)
I20260923 17:28:05.584643 8887 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:35245
I20260923 17:28:05.588393 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:05.607990 9078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.594669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52040 (local address 127.8.29.254:39547)
0923 17:28:05.594858 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.594864 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.595351 (+ 487us) server_negotiation.cc:407] Connection header received
0923 17:28:05.595489 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.595494 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.595602 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.595713 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:05.597471 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.598629 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.600151 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.600581 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.600695 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.602994 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:05.602999 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:05.603003 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:05.603037 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:05.605447 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.606068 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.606074 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.606078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.606204 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.606575 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.606580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.606583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.606900 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:05.607124 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.607479 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.607697 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:28:05.614634 8887 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52040:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:05.621403 8887 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:05.626397 9076 heartbeater.cc:503] Master 127.8.29.254:39547 was elected leader, sending a full tablet report...
I20260923 17:28:05.643775 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.634395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33870 (local address 127.8.29.194:36329)
0923 17:28:05.635355 (+ 960us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.635364 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.635393 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:28:05.635500 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.635521 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.635812 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.636614 (+ 802us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:05.636826 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.638369 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.641259 (+ 2890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.642718 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.642844 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.643016 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.643286 (+ 270us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.643454 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":828,"thread_start_us":766,"threads_started":1}
I20260923 17:28:05.643903 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.635035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41834 (local address 127.8.29.193:38895)
0923 17:28:05.635943 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.635952 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.635980 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:28:05.636080 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.636087 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.636308 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.636487 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:05.637120 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.638156 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.640793 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.642122 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.642254 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.642509 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.643461 (+ 952us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.643624 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":749,"thread_start_us":630,"threads_started":1}
I20260923 17:28:05.643875 9355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.633462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:43069 (local address 127.0.0.1:33210)
0923 17:28:05.634751 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:05.634793 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:05.634986 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:05.636744 (+ 1758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:05.636750 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:05.636770 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:05.637121 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.637132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.639150 (+ 2018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.639155 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.641254 (+ 2099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.641266 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.643156 (+ 1890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.643162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.643314 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.643345 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:28:05.643413 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 17:28:05.643536 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1072,"mutex_wait_us":1,"thread_start_us":289,"threads_started":1}
I20260923 17:28:05.644201 9359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.634047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:36329 (local address 127.0.0.1:33870)
0923 17:28:05.635092 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:05.635108 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:05.635207 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:05.636058 (+ 851us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:05.636067 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:05.636099 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:05.636404 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.636421 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.638552 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.638558 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.641005 (+ 2447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.641021 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.642926 (+ 1905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.642933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.643099 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.643132 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:28:05.643223 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 17:28:05.643354 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"mutex_wait_us":34,"thread_start_us":318,"threads_started":1}
I20260923 17:28:05.644328 9357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.634598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:38895 (local address 127.0.0.1:41834)
0923 17:28:05.635318 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:05.635332 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:05.635413 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:05.636486 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:05.636491 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:05.636515 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:05.636911 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.636924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.638310 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.638314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.640599 (+ 2285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.640611 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.642626 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.642633 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.642832 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.642893 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 17:28:05.643040 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 17:28:05.643195 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":142,"thread_start_us":290,"threads_started":1}
I20260923 17:28:05.644937 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.633824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33210 (local address 127.8.29.195:43069)
0923 17:28:05.635990 (+ 2166us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.635998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.636034 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:28:05.636116 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.636122 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.636294 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.636487 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:05.637292 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.638982 (+ 1690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.641405 (+ 2423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.642679 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.642794 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.643022 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.644472 (+ 1450us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.644664 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2020,"thread_start_us":2229,"threads_started":1}
I20260923 17:28:05.648025 9145 tablet_service.cc:1467] Processing CreateTablet for tablet 028c56612a95425891326c647e74c306 (DEFAULT_TABLE table=test-table [id=3a3fe9086723442a8deb5bac2c64bbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:05.648449 9278 tablet_service.cc:1467] Processing CreateTablet for tablet 028c56612a95425891326c647e74c306 (DEFAULT_TABLE table=test-table [id=3a3fe9086723442a8deb5bac2c64bbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:05.648449 9011 tablet_service.cc:1467] Processing CreateTablet for tablet 028c56612a95425891326c647e74c306 (DEFAULT_TABLE table=test-table [id=3a3fe9086723442a8deb5bac2c64bbf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:05.649718 9145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028c56612a95425891326c647e74c306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:05.649798 9011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028c56612a95425891326c647e74c306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:05.649811 9278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 028c56612a95425891326c647e74c306. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:05.662735 9361 tablet_bootstrap.cc:492] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864: Bootstrap starting.
I20260923 17:28:05.666644 9362 tablet_bootstrap.cc:492] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a: Bootstrap starting.
I20260923 17:28:05.669064 9361 tablet_bootstrap.cc:654] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:05.671362 9361 log.cc:824] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:05.671808 9363 tablet_bootstrap.cc:492] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf: Bootstrap starting.
I20260923 17:28:05.673324 9362 tablet_bootstrap.cc:654] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:05.675212 9363 tablet_bootstrap.cc:654] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:05.675227 9362 log.cc:824] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:05.675347 9361 tablet_bootstrap.cc:492] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864: No bootstrap required, opened a new log
I20260923 17:28:05.675618 9361 ts_tablet_manager.cc:1397] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20260923 17:28:05.676335 9363 log.cc:824] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:05.679128 9362 tablet_bootstrap.cc:492] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a: No bootstrap required, opened a new log
I20260923 17:28:05.679527 9362 ts_tablet_manager.cc:1397] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20260923 17:28:05.685068 9363 tablet_bootstrap.cc:492] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf: No bootstrap required, opened a new log
I20260923 17:28:05.685361 9363 ts_tablet_manager.cc:1397] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260923 17:28:05.685822 9361 raft_consensus.cc:348] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } }
I20260923 17:28:05.686295 9361 raft_consensus.cc:374] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:05.686390 9361 raft_consensus.cc:729] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa65ecd800dc408f9c614de091269864, State: Initialized, Role: FOLLOWER
I20260923 17:28:05.687077 9361 consensus_queue.cc:260] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } }
I20260923 17:28:05.689967 9210 heartbeater.cc:503] Master 127.8.29.254:39547 was elected leader, sending a full tablet report...
I20260923 17:28:05.690662 9361 ts_tablet_manager.cc:1428] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864: Time spent starting tablet: real 0.015s user 0.015s sys 0.001s
I20260923 17:28:05.691452 9362 raft_consensus.cc:348] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } }
I20260923 17:28:05.691892 9362 raft_consensus.cc:374] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:05.692001 9362 raft_consensus.cc:729] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f01bc01166ee4993aee95fd868f9039a, State: Initialized, Role: FOLLOWER
I20260923 17:28:05.692638 9362 consensus_queue.cc:260] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } }
I20260923 17:28:05.693745 9343 heartbeater.cc:503] Master 127.8.29.254:39547 was elected leader, sending a full tablet report...
I20260923 17:28:05.694418 9362 ts_tablet_manager.cc:1428] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260923 17:28:05.695161 9363 raft_consensus.cc:348] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } }
I20260923 17:28:05.695596 9363 raft_consensus.cc:374] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:05.695700 9363 raft_consensus.cc:729] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6becb83013584aec90bdee9ac2b3beaf, State: Initialized, Role: FOLLOWER
I20260923 17:28:05.696251 9363 consensus_queue.cc:260] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } }
I20260923 17:28:05.697861 9363 ts_tablet_manager.cc:1428] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf: Time spent starting tablet: real 0.012s user 0.012s sys 0.003s
W20260923 17:28:05.809535 9344 tablet.cc:2369] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:05.822029 9369 raft_consensus.cc:482] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:05.822338 9369 raft_consensus.cc:504] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } }
I20260923 17:28:05.824299 9369 leader_election.cc:288] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f01bc01166ee4993aee95fd868f9039a (127.8.29.195:43069), aa65ecd800dc408f9c614de091269864 (127.8.29.194:36329)
W20260923 17:28:05.827138 9211 tablet.cc:2369] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:05.832405 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.824746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:57911 (local address 127.8.29.194:36329)
0923 17:28:05.824979 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.824986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.825659 (+ 673us) server_negotiation.cc:407] Connection header received
0923 17:28:05.825823 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.825829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.825900 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.825977 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:05.826605 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.827485 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.830107 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.831336 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.831498 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.831570 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.832014 (+ 444us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.832145 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 17:28:05.832427 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.824739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:43619 (local address 127.8.29.195:43069)
0923 17:28:05.824940 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.824947 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.825722 (+ 775us) server_negotiation.cc:407] Connection header received
0923 17:28:05.825872 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.825876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.825959 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.826048 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:05.826715 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.827704 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.830107 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.831338 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.831494 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.831570 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.832014 (+ 444us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.832145 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:28:05.832634 9371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.824729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:43069 (local address 127.8.29.193:43619)
0923 17:28:05.825599 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:05.825619 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:05.825749 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:05.826121 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:05.826126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:05.826159 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:05.826565 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.826577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.827837 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.827842 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.829938 (+ 2096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.829952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.831582 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.831589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.831803 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.831864 (+ 61us) client_negotiation.cc:769] Sending connection context
0923 17:28:05.831958 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:28:05.832152 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":93,"thread_start_us":499,"threads_started":1}
I20260923 17:28:05.832913 9370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.824574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:36329 (local address 127.8.29.193:57911)
0923 17:28:05.825543 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:05.825559 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:05.825683 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:05.826017 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:05.826022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:05.826046 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:05.826425 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.826435 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.827620 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.827626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.829940 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:05.829952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.831588 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.831593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:05.831797 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.831871 (+ 74us) client_negotiation.cc:769] Sending connection context
0923 17:28:05.831958 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 17:28:05.832152 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":857,"mutex_wait_us":136,"thread_start_us":633,"threads_started":1}
I20260923 17:28:05.833725 9298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028c56612a95425891326c647e74c306" candidate_uuid: "6becb83013584aec90bdee9ac2b3beaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01bc01166ee4993aee95fd868f9039a" is_pre_election: true
I20260923 17:28:05.833781 9165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028c56612a95425891326c647e74c306" candidate_uuid: "6becb83013584aec90bdee9ac2b3beaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa65ecd800dc408f9c614de091269864" is_pre_election: true
I20260923 17:28:05.834306 9165 raft_consensus.cc:2454] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6becb83013584aec90bdee9ac2b3beaf in term 0.
I20260923 17:28:05.834306 9298 raft_consensus.cc:2454] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6becb83013584aec90bdee9ac2b3beaf in term 0.
I20260923 17:28:05.835064 8965 leader_election.cc:302] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6becb83013584aec90bdee9ac2b3beaf, aa65ecd800dc408f9c614de091269864; no voters:
I20260923 17:28:05.835543 9369 raft_consensus.cc:2781] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:05.835693 9369 raft_consensus.cc:482] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:05.835793 9369 raft_consensus.cc:3037] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:05.837957 9369 raft_consensus.cc:504] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } }
I20260923 17:28:05.838632 9369 leader_election.cc:288] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [CANDIDATE]: Term 1 election: Requested vote from peers f01bc01166ee4993aee95fd868f9039a (127.8.29.195:43069), aa65ecd800dc408f9c614de091269864 (127.8.29.194:36329)
I20260923 17:28:05.839165 9165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028c56612a95425891326c647e74c306" candidate_uuid: "6becb83013584aec90bdee9ac2b3beaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa65ecd800dc408f9c614de091269864"
I20260923 17:28:05.839183 9298 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "028c56612a95425891326c647e74c306" candidate_uuid: "6becb83013584aec90bdee9ac2b3beaf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f01bc01166ee4993aee95fd868f9039a"
I20260923 17:28:05.839370 9298 raft_consensus.cc:3037] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:05.839370 9165 raft_consensus.cc:3037] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:05.841624 9165 raft_consensus.cc:2454] T 028c56612a95425891326c647e74c306 P aa65ecd800dc408f9c614de091269864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6becb83013584aec90bdee9ac2b3beaf in term 1.
I20260923 17:28:05.841626 9298 raft_consensus.cc:2454] T 028c56612a95425891326c647e74c306 P f01bc01166ee4993aee95fd868f9039a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6becb83013584aec90bdee9ac2b3beaf in term 1.
I20260923 17:28:05.842049 8965 leader_election.cc:302] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6becb83013584aec90bdee9ac2b3beaf, aa65ecd800dc408f9c614de091269864; no voters:
I20260923 17:28:05.842394 9369 raft_consensus.cc:2781] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:05.842875 9369 raft_consensus.cc:686] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [term 1 LEADER]: Becoming Leader. State: Replica: 6becb83013584aec90bdee9ac2b3beaf, State: Running, Role: LEADER
I20260923 17:28:05.843358 9369 consensus_queue.cc:237] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } }
W20260923 17:28:05.847225 9077 tablet.cc:2369] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:05.848224 8887 catalog_manager.cc:5495] T 028c56612a95425891326c647e74c306 P 6becb83013584aec90bdee9ac2b3beaf reported cstate change: term changed from 0 to 1, leader changed from <none> to 6becb83013584aec90bdee9ac2b3beaf (127.8.29.193). New cstate: current_term: 1 leader_uuid: "6becb83013584aec90bdee9ac2b3beaf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f01bc01166ee4993aee95fd868f9039a" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43069 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6becb83013584aec90bdee9ac2b3beaf" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38895 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa65ecd800dc408f9c614de091269864" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36329 } health_report { overall_health: UNKNOWN } } }
Sep 23 17:28:05 dist-test-slave-p7jt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:05.892246 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.877285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41844 (local address 127.8.29.193:38895)
0923 17:28:05.877562 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.877570 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.877971 (+ 401us) server_negotiation.cc:407] Connection header received
0923 17:28:05.878153 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.878159 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.878242 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.878338 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:05.879983 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.880754 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.882071 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.882548 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.882675 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.886776 (+ 4101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:05.886793 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:05.886809 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:05.886846 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:05.889472 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.890136 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.890144 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.890147 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.890300 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.890714 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.890720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.890723 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.891083 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:05.891353 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.891678 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.891922 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
Sep 23 17:28:05 dist-test-slave-p7jt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184485, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:28:05 dist-test-slave-p7jt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184485, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:05.936719 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.922544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41852 (local address 127.8.29.193:38895)
0923 17:28:05.922794 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.922802 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.923154 (+ 352us) server_negotiation.cc:407] Connection header received
0923 17:28:05.923362 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.923366 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.923445 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.923547 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:05.925031 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.925894 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.927342 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.927773 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.927910 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.931813 (+ 3903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:05.931818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:05.931821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:05.931860 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:05.934168 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.934856 (+ 688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.934861 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.934863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.934996 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.935385 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.935388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.935390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.935667 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:05.935930 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.936180 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.936396 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
Sep 23 17:28:05 dist-test-slave-p7jt krb5kdc[8842](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:28:05 dist-test-slave-p7jt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:05.979622 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.965350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41856 (local address 127.8.29.193:38895)
0923 17:28:05.965538 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.965566 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.965864 (+ 298us) server_negotiation.cc:407] Connection header received
0923 17:28:05.966040 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.966046 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.966128 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.966241 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:05.967644 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.968421 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.969673 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.970207 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.970330 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.974333 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:05.974338 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:05.974342 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:05.974380 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:05.977101 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.977768 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.977772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.977774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.977904 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.978284 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.978288 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.978290 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.978647 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:05.978931 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.979173 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.979347 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 17:28:05.981350 8991 tablet_service.cc:2908] Rejecting scan request for tablet 028c56612a95425891326c647e74c306: Uninitialized: clean time has not yet been initialized
I20260923 17:28:05.985074 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 8948
I20260923 17:28:05.998695 9360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:05.986046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41872 (local address 127.8.29.193:38895)
0923 17:28:05.986329 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:05.986336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:05.986529 (+ 193us) server_negotiation.cc:407] Connection header received
0923 17:28:05.986701 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:05.986705 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:05.986771 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:05.986862 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:05.988248 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.988984 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.990209 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:05.990666 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:05.990784 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:05.993081 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:05.993086 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:05.993089 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:05.993126 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:05.995831 (+ 2705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.996630 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.996634 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.996636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.996792 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:05.997182 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:05.997186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:05.997188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:05.997467 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:05.997709 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:05.998034 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 17:28:05.998307 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260923 17:28:05.999421 9071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:06.095053 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 8948
I20260923 17:28:06.109123 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 9082
Sep 23 17:28:06 dist-test-slave-p7jt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:28:06.123677 9358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:06.109983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33880 (local address 127.8.29.194:36329)
0923 17:28:06.110144 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:06.110148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:06.110166 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:06.110224 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:06.110227 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:06.110289 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:06.110376 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:06.111586 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:06.112380 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:06.113604 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:06.114014 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:06.114116 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:06.118155 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:06.118173 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:06.118189 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:06.118230 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:06.120941 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:06.121674 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:06.121682 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:06.121684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:06.121810 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:06.122124 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:06.122127 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:06.122129 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:06.122721 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:06.122941 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:06.123244 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 17:28:06.123417 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:28:06.124401 9205 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:06.211704 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 9082
I20260923 17:28:06.225126 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 9215
Sep 23 17:28:06 dist-test-slave-p7jt krb5kdc[8842](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184485, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:28:06.240124 9356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:06.225974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.8.29.195:43069)
0923 17:28:06.226167 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:06.226173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:06.226195 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:28:06.226316 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:06.226320 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:06.226385 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:06.226499 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:06.227727 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:06.228422 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:06.229683 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:06.230083 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:06.230193 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:06.234305 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:06.234322 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:06.234341 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:06.234386 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:06.237125 (+ 2739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:06.237907 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:06.237917 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:06.237919 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:06.238090 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:06.238575 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:06.238580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:06.238583 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:06.239062 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:06.239283 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:06.239547 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 17:28:06.239728 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:28:06.240921 9338 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:06.327342 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 9215
I20260923 17:28:06.340035 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 8855
I20260923 17:28:06.341241 8917 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:06.447784 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 8855
2026-09-23T17:28:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4062 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:06 dist-test-slave-p7jt krb5kdc[9391](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:06 dist-test-slave-p7jt krb5kdc[9391](info): set up 2 sockets
Sep 23 17:28:06 dist-test-slave-p7jt krb5kdc[9391](info): commencing operation
krb5kdc: starting...
W20260923 17:28:07.450559 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.964s user 0.007s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:07 dist-test-slave-p7jt krb5kdc[9391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:28:07.626340 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:46175
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:46175
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:07.945137 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:07.945441 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:07.945554 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:07.952780 9404 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:07.952890 9404 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:28:07.952930 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:07.953013 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:07.953081 9404 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:07.953174 9404 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:07.963680 9404 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:46175
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:46175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:07.965059 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:07.966522 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:07.978688 9409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:07.978716 9410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:07.978756 9412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:07.979285 9404 server_base.cc:1034] running on GCE node
I20260923 17:28:07.980150 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:07.981698 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:07.982909 9404 hybrid_clock.cc:648] HybridClock initialized: now 1790184487982841 us; error 48 us; skew 500 ppm
I20260923 17:28:07.983290 9404 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:07 dist-test-slave-p7jt krb5kdc[9391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184487, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:07.986490 9404 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:07.988104 9404 webserver.cc:460] Webserver started at http://127.8.29.254:42825/ using document root <none> and password file <none>
I20260923 17:28:07.988826 9404 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:07.988919 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:07.989223 9404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:07.992254 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "1a52c9f63e784608b51fba38e62b756c"
format_stamp: "Formatted at 2026-09-23 17:28:07 on dist-test-slave-p7jt"
server_key: "9267476c4c8e390708cf20a6e39cb37c"
server_key_iv: "fb45eb638e32701afabd29e257725e5a"
server_key_version: "encryptionkey@0"
I20260923 17:28:07.993147 9404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "1a52c9f63e784608b51fba38e62b756c"
format_stamp: "Formatted at 2026-09-23 17:28:07 on dist-test-slave-p7jt"
server_key: "9267476c4c8e390708cf20a6e39cb37c"
server_key_iv: "fb45eb638e32701afabd29e257725e5a"
server_key_version: "encryptionkey@0"
I20260923 17:28:07.998540 9404 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 17:28:08.002338 9419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:08.003899 9404 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:08.004073 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "1a52c9f63e784608b51fba38e62b756c"
format_stamp: "Formatted at 2026-09-23 17:28:07 on dist-test-slave-p7jt"
server_key: "9267476c4c8e390708cf20a6e39cb37c"
server_key_iv: "fb45eb638e32701afabd29e257725e5a"
server_key_version: "encryptionkey@0"
I20260923 17:28:08.004300 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:08.029376 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:08.033463 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:08.033725 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:08.054780 9404 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:46175
I20260923 17:28:08.054751 9480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:46175 every 8 connection(s)
I20260923 17:28:08.056258 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:08.060067 9481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:08.063298 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 9404
I20260923 17:28:08.063521 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:08.064154 8311 external_mini_cluster.cc:1428] Setting key b84d6d4666a4132d22e50a8cc9b69956
I20260923 17:28:08.072899 9481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c: Bootstrap starting.
I20260923 17:28:08.077055 9481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:08.078118 9481 log.cc:824] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:28:08 dist-test-slave-p7jt krb5kdc[9391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184487, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:08.081374 9481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c: No bootstrap required, opened a new log
I20260923 17:28:08.087904 9484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:08.066133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.8.29.254:46175)
0923 17:28:08.066943 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:08.066955 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:08.066997 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:28:08.068662 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:08.068686 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:08.069151 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:08.069520 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:08.071391 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.072791 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.074674 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.075812 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.076005 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:08.080167 (+ 4162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:08.080177 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:08.080195 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:08.080241 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:08.083497 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:08.084311 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:08.084320 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:08.084330 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:08.084573 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:08.084973 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:08.084977 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:08.084979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:08.085530 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:08.085883 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:08.086318 (+ 435us) server_negotiation.cc:299] Negotiation successful
0923 17:28:08.086808 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":390,"threads_started":1}
I20260923 17:28:08.090114 9481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a52c9f63e784608b51fba38e62b756c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46175 } }
I20260923 17:28:08.090521 9481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:08.090624 9481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a52c9f63e784608b51fba38e62b756c, State: Initialized, Role: FOLLOWER
I20260923 17:28:08.091267 9481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a52c9f63e784608b51fba38e62b756c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46175 } }
I20260923 17:28:08.091499 9481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:08.091617 9481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:08.091771 9481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:08.093942 9481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a52c9f63e784608b51fba38e62b756c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46175 } }
I20260923 17:28:08.094485 9481 leader_election.cc:302] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a52c9f63e784608b51fba38e62b756c; no voters:
I20260923 17:28:08.095119 9481 leader_election.cc:288] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:08.095252 9486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:08.095891 9486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [term 1 LEADER]: Becoming Leader. State: Replica: 1a52c9f63e784608b51fba38e62b756c, State: Running, Role: LEADER
I20260923 17:28:08.096612 9486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a52c9f63e784608b51fba38e62b756c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46175 } }
I20260923 17:28:08.097637 9481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:08.099313 9488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a52c9f63e784608b51fba38e62b756c. Latest consensus state: current_term: 1 leader_uuid: "1a52c9f63e784608b51fba38e62b756c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a52c9f63e784608b51fba38e62b756c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46175 } } }
I20260923 17:28:08.099641 9487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a52c9f63e784608b51fba38e62b756c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a52c9f63e784608b51fba38e62b756c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46175 } } }
I20260923 17:28:08.099867 9488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:08.099885 9487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:08.102335 9493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:08.107442 9493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:08.117089 9493 catalog_manager.cc:1347] Generated new cluster ID: f7d8fd5d2133445a87cf1f1d8f3b7958
I20260923 17:28:08.117221 9493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:08.138074 9493 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:08.140043 9493 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:08.154291 9493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c: Generated new TSK 0
I20260923 17:28:08.155241 9493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:28:08.212136 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:46175
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:08.560444 9507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:08.560734 9507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:08.560851 9507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:08.568609 9507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:08.568766 9507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:08.568902 9507 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:28:08.569002 9507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:08.579164 9507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.29.254:46175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:08.580438 9507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:08.581746 9507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:08.595398 9512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:08.595374 9513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:08.595798 9515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:08.596222 9507 server_base.cc:1034] running on GCE node
I20260923 17:28:08.596824 9507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:08.597847 9507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:08.599079 9507 hybrid_clock.cc:648] HybridClock initialized: now 1790184488599014 us; error 46 us; skew 500 ppm
I20260923 17:28:08.599416 9507 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:08 dist-test-slave-p7jt krb5kdc[9391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:08.602890 9507 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:28:08.604447 9507 webserver.cc:460] Webserver started at http://127.8.29.193:42903/ using document root <none> and password file <none>
I20260923 17:28:08.605188 9507 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:08.605294 9507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:08.605628 9507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:08.609126 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "0f6378567c064c83a7431db623cade49"
format_stamp: "Formatted at 2026-09-23 17:28:08 on dist-test-slave-p7jt"
server_key: "d15ffbe163581155bf0e34486fcd7ed0"
server_key_iv: "668c80845e6304a8364d3577303d8d02"
server_key_version: "encryptionkey@0"
I20260923 17:28:08.610158 9507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "0f6378567c064c83a7431db623cade49"
format_stamp: "Formatted at 2026-09-23 17:28:08 on dist-test-slave-p7jt"
server_key: "d15ffbe163581155bf0e34486fcd7ed0"
server_key_iv: "668c80845e6304a8364d3577303d8d02"
server_key_version: "encryptionkey@0"
I20260923 17:28:08.615892 9507 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 17:28:08.619627 9522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:08.621062 9507 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:28:08.621210 9507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "0f6378567c064c83a7431db623cade49"
format_stamp: "Formatted at 2026-09-23 17:28:08 on dist-test-slave-p7jt"
server_key: "d15ffbe163581155bf0e34486fcd7ed0"
server_key_iv: "668c80845e6304a8364d3577303d8d02"
server_key_version: "encryptionkey@0"
I20260923 17:28:08.621404 9507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:08.654002 9507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:08.658749 9507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:08.659027 9507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:08.661793 9507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:08.661882 9507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:08.662024 9507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:08.662088 9507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:28:08 dist-test-slave-p7jt krb5kdc[9391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184488, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:08.687317 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:08.662597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:40773 (local address 127.8.29.254:46175)
0923 17:28:08.663053 (+ 456us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:08.663060 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:08.664483 (+ 1423us) server_negotiation.cc:407] Connection header received
0923 17:28:08.667158 (+ 2675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:08.667166 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:08.667274 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:08.667413 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:08.669638 (+ 2225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.670732 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.673093 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.673612 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.673745 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:08.678886 (+ 5141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:08.678894 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:08.678900 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:08.678954 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:08.682334 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:08.683673 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:08.683679 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:08.683683 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:08.683869 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:08.684544 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:08.684549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:08.684553 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:08.684941 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:08.685115 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:08.686692 (+ 1577us) server_negotiation.cc:299] Negotiation successful
0923 17:28:08.686932 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":223,"threads_started":1}
I20260923 17:28:08.688575 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:08.663383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46175 (local address 127.8.29.193:40773)
0923 17:28:08.664261 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:08.664319 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:08.666835 (+ 2516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:08.667706 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:08.667720 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:08.668498 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:08.669431 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:08.669457 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.670952 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.670961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:08.672863 (+ 1902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:08.672884 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.673879 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.673887 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:08.674148 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:08.675037 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:08.675074 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:08.678507 (+ 3433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:08.682558 (+ 4051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:08.682575 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:08.682601 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:08.683456 (+ 855us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:08.684024 (+ 568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:08.684030 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:08.684037 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:08.684376 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:08.685110 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:08.685120 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:08.685700 (+ 580us) client_negotiation.cc:769] Sending connection context
0923 17:28:08.686076 (+ 376us) client_negotiation.cc:240] Negotiation successful
0923 17:28:08.686804 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":357,"threads_started":1}
I20260923 17:28:08.699908 9507 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:41559
I20260923 17:28:08.699946 9642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:41559 every 8 connection(s)
I20260923 17:28:08.701308 9507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:08.702292 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 9507
I20260923 17:28:08.702562 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:08.703338 8311 external_mini_cluster.cc:1428] Setting key fb75d1cb49723b7f95241e6245e754fa
I20260923 17:28:08.705313 9643 heartbeater.cc:347] Connected to a master server at 127.8.29.254:46175
I20260923 17:28:08.705586 9643 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:08.706321 9643 heartbeater.cc:511] Master 127.8.29.254:46175 requested a full tablet report, sending...
I20260923 17:28:08.708602 9436 ts_manager.cc:194] Registered new tserver with Master: 0f6378567c064c83a7431db623cade49 (127.8.29.193:41559)
I20260923 17:28:08.710629 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:08.696992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:53601 (local address 127.8.29.254:46175)
0923 17:28:08.697182 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:08.697188 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:08.697208 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:28:08.697422 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:08.697428 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:08.697514 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:08.697608 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:08.698934 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.700011 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.701427 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.702004 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.702140 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:08.704260 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:08.704267 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:08.704271 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:08.704312 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:08.707622 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:08.708435 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:08.708441 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:08.708444 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:08.708614 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:08.709025 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:08.709029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:08.709042 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:08.709421 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:08.709562 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:08.710016 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 17:28:08.710233 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:28:08.710682 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:08.696382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46175 (local address 127.8.29.193:53601)
0923 17:28:08.697116 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:08.697136 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:08.697283 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:08.697799 (+ 516us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:08.697806 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:08.698308 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:08.698718 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:08.698737 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.700215 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.700218 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:08.701263 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:08.701276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:08.702187 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:08.702193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:08.702377 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:08.703176 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:08.703201 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:08.704048 (+ 847us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:08.707778 (+ 3730us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:08.707784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:08.707788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:08.708295 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:08.708744 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:08.708749 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:08.708752 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:08.708920 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:08.709555 (+ 635us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:08.709560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:08.709746 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 17:28:08.709896 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 17:28:08.710230 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":411,"threads_started":1}
I20260923 17:28:08.710619 9436 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:40773
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:28:08.746511 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:46175
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:09.064208 9647 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:09.064492 9647 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:09.064625 9647 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:09.072031 9647 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:09.072165 9647 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:09.072273 9647 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:28:09.072383 9647 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:09.083750 9647 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.29.254:46175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:09.085151 9647 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:09.086652 9647 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:09.099597 9652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:09.099649 9655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:09.099601 9653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:09.100536 9647 server_base.cc:1034] running on GCE node
I20260923 17:28:09.101171 9647 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:09.102161 9647 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:09.103402 9647 hybrid_clock.cc:648] HybridClock initialized: now 1790184489103278 us; error 89 us; skew 500 ppm
I20260923 17:28:09.103704 9647 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:09 dist-test-slave-p7jt krb5kdc[9391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:09.110267 9647 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:28:09.111917 9647 webserver.cc:460] Webserver started at http://127.8.29.194:37563/ using document root <none> and password file <none>
I20260923 17:28:09.112612 9647 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:09.112720 9647 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:09.113029 9647 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:09.115907 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "1fb1cd1625774ada9d54bf3fcc25df0b"
format_stamp: "Formatted at 2026-09-23 17:28:09 on dist-test-slave-p7jt"
server_key: "5b2da3d07ec20bb25b49169d6a12eb39"
server_key_iv: "9059887670cde155ce3257c443abaab7"
server_key_version: "encryptionkey@0"
I20260923 17:28:09.116715 9647 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "1fb1cd1625774ada9d54bf3fcc25df0b"
format_stamp: "Formatted at 2026-09-23 17:28:09 on dist-test-slave-p7jt"
server_key: "5b2da3d07ec20bb25b49169d6a12eb39"
server_key_iv: "9059887670cde155ce3257c443abaab7"
server_key_version: "encryptionkey@0"
I20260923 17:28:09.122345 9647 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:28:09.126235 9662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:09.127737 9647 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:09.127887 9647 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "1fb1cd1625774ada9d54bf3fcc25df0b"
format_stamp: "Formatted at 2026-09-23 17:28:09 on dist-test-slave-p7jt"
server_key: "5b2da3d07ec20bb25b49169d6a12eb39"
server_key_iv: "9059887670cde155ce3257c443abaab7"
server_key_version: "encryptionkey@0"
I20260923 17:28:09.128084 9647 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:09.155159 9647 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:09.159569 9647 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:09.159817 9647 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:09.162685 9647 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:09.162782 9647 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:09.162932 9647 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:09.163005 9647 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:28:09 dist-test-slave-p7jt krb5kdc[9391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:09.186151 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.163336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:58467 (local address 127.8.29.254:46175)
0923 17:28:09.163552 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.163558 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.165022 (+ 1464us) server_negotiation.cc:407] Connection header received
0923 17:28:09.168048 (+ 3026us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.168055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.168152 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.168272 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:09.170704 (+ 2432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.171668 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.174609 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.175237 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.175339 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.180001 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:09.180006 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:09.180009 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:09.180045 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:09.182606 (+ 2561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.183682 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.183687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.183690 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.183832 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.184378 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.184383 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.184387 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.184679 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:09.184822 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.185580 (+ 758us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.185804 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:28:09.187397 9671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.163911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46175 (local address 127.8.29.194:58467)
0923 17:28:09.164807 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:09.164865 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:09.167581 (+ 2716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:09.168620 (+ 1039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:09.168637 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:09.169489 (+ 852us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:09.170380 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.170414 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.171838 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.171844 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.174370 (+ 2526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.174392 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.175487 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.175495 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.175803 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.176573 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:09.176605 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:09.179755 (+ 3150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:09.182754 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:09.182771 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:09.182790 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:09.183498 (+ 708us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:09.183973 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:09.183978 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:09.183982 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:09.184249 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:09.184817 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:09.184825 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:09.185197 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 17:28:09.185437 (+ 240us) client_negotiation.cc:240] Negotiation successful
0923 17:28:09.185991 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":347,"threads_started":1}
I20260923 17:28:09.209661 9729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.194248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46175 (local address 127.8.29.194:45331)
0923 17:28:09.194866 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:09.194888 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:09.195015 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:09.195346 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:09.195348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:09.195789 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:09.196063 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.196072 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.197614 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.197621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.199129 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.199147 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.200255 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.200260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.200429 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.201151 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:09.201171 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:09.201960 (+ 789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:09.206238 (+ 4278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:09.206244 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:09.206252 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:09.206875 (+ 623us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:09.207335 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:09.207342 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:09.207346 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:09.207559 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:09.208315 (+ 756us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:09.208320 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:09.208614 (+ 294us) client_negotiation.cc:769] Sending connection context
0923 17:28:09.208813 (+ 199us) client_negotiation.cc:240] Negotiation successful
0923 17:28:09.209207 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":412,"threads_started":1}
I20260923 17:28:09.209817 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.194590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:45331 (local address 127.8.29.254:46175)
0923 17:28:09.194818 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.194825 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.194968 (+ 143us) server_negotiation.cc:407] Connection header received
0923 17:28:09.195124 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.195130 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.195213 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.195315 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:09.196188 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.197379 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.199309 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.200025 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.200167 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.202240 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:09.202248 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:09.202254 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:09.202333 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:09.206059 (+ 3726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.207017 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.207024 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.207028 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.207194 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.207733 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.207739 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.207741 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.208203 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:09.208347 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.209291 (+ 944us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.209528 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:28:09.220718 9647 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:42327
I20260923 17:28:09.220731 9781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:42327 every 8 connection(s)
I20260923 17:28:09.221959 9647 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:09.224282 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 9647
I20260923 17:28:09.224538 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:09.224509 9782 heartbeater.cc:347] Connected to a master server at 127.8.29.254:46175
I20260923 17:28:09.224774 9782 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:09.225004 8311 external_mini_cluster.cc:1428] Setting key 710789fa54e8219871633cb74038c113
I20260923 17:28:09.225538 9782 heartbeater.cc:511] Master 127.8.29.254:46175 requested a full tablet report, sending...
I20260923 17:28:09.227214 9435 ts_manager.cc:194] Registered new tserver with Master: 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327)
I20260923 17:28:09.228116 9435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:58467
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:28:09.266338 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:46175
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:41711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:09.587718 9786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:09.588008 9786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:09.588132 9786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:09.595913 9786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:09.596024 9786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:09.596146 9786 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:28:09.596261 9786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:09.606401 9786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:41711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.29.254:46175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:09.607844 9786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:09.609344 9786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:09.623754 9791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:09.623819 9794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:09.623795 9792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:09.625079 9786 server_base.cc:1034] running on GCE node
I20260923 17:28:09.625761 9786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:09.626876 9786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:09.628131 9786 hybrid_clock.cc:648] HybridClock initialized: now 1790184489628040 us; error 59 us; skew 500 ppm
I20260923 17:28:09.628435 9786 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:09 dist-test-slave-p7jt krb5kdc[9391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:09.631733 9786 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:28:09.633494 9786 webserver.cc:460] Webserver started at http://127.8.29.195:38281/ using document root <none> and password file <none>
I20260923 17:28:09.634220 9786 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:09.634308 9786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:09.634650 9786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:09.637619 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "1136da8d8da14bc89cab48bdf49b9d19"
format_stamp: "Formatted at 2026-09-23 17:28:09 on dist-test-slave-p7jt"
server_key: "d35f1f7167b3048115b95e2a778bc3ed"
server_key_iv: "050dff70998b5c878fda84dcd92c3635"
server_key_version: "encryptionkey@0"
I20260923 17:28:09.638672 9786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "1136da8d8da14bc89cab48bdf49b9d19"
format_stamp: "Formatted at 2026-09-23 17:28:09 on dist-test-slave-p7jt"
server_key: "d35f1f7167b3048115b95e2a778bc3ed"
server_key_iv: "050dff70998b5c878fda84dcd92c3635"
server_key_version: "encryptionkey@0"
I20260923 17:28:09.644531 9786 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:09.648519 9801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:09.650117 9786 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:09.650274 9786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "1136da8d8da14bc89cab48bdf49b9d19"
format_stamp: "Formatted at 2026-09-23 17:28:09 on dist-test-slave-p7jt"
server_key: "d35f1f7167b3048115b95e2a778bc3ed"
server_key_iv: "050dff70998b5c878fda84dcd92c3635"
server_key_version: "encryptionkey@0"
I20260923 17:28:09.650504 9786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:09.671020 9786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:09.675438 9786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:09.675665 9786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:09.678355 9786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:09.678442 9786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:09.678633 9786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:09.678696 9786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:28:09 dist-test-slave-p7jt krb5kdc[9391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184489, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:09.702430 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.679183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:36837 (local address 127.8.29.254:46175)
0923 17:28:09.679376 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.679383 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.680884 (+ 1501us) server_negotiation.cc:407] Connection header received
0923 17:28:09.683303 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.683308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.683410 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.683530 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:09.686077 (+ 2547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.687026 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.689950 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.690557 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.690677 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.695264 (+ 4587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:09.695272 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:09.695278 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:09.695327 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:09.698677 (+ 3350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.699617 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.699624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.699628 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.699804 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.700313 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.700318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.700322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.700696 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:09.700844 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.701760 (+ 916us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.702028 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:28:09.703708 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.679725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46175 (local address 127.8.29.195:36837)
0923 17:28:09.680661 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:09.680719 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:09.682999 (+ 2280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:09.683903 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:09.683921 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:09.684833 (+ 912us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:09.685845 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.685876 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.687221 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.687231 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.689758 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.689780 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.690821 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.690830 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.691071 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.691814 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:09.691835 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:09.694592 (+ 2757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:09.698824 (+ 4232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:09.698831 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:09.698845 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:09.699470 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:09.699947 (+ 477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:09.699952 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:09.699955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:09.700203 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:09.700851 (+ 648us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:09.700859 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:09.701300 (+ 441us) client_negotiation.cc:769] Sending connection context
0923 17:28:09.701604 (+ 304us) client_negotiation.cc:240] Negotiation successful
0923 17:28:09.702133 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":408,"threads_started":1}
I20260923 17:28:09.714634 9643 heartbeater.cc:503] Master 127.8.29.254:46175 was elected leader, sending a full tablet report...
I20260923 17:28:09.732707 9881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.717725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46175 (local address 127.8.29.195:53057)
0923 17:28:09.718375 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:09.718402 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:09.718624 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:09.719255 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:09.719260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:09.719941 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:09.720531 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.720547 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.721903 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.721909 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.723702 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.723721 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.724756 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.724763 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.724939 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.725652 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:09.725675 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:09.726552 (+ 877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:09.729787 (+ 3235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:09.729794 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:09.729797 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:09.730229 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:09.730714 (+ 485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:09.730720 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:09.730724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:09.730901 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:09.731693 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:09.731699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:09.731885 (+ 186us) client_negotiation.cc:769] Sending connection context
0923 17:28:09.732032 (+ 147us) client_negotiation.cc:240] Negotiation successful
0923 17:28:09.732357 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":345,"threads_started":1}
I20260923 17:28:09.732846 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.717784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:53057 (local address 127.8.29.254:46175)
0923 17:28:09.717997 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.718004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.718528 (+ 524us) server_negotiation.cc:407] Connection header received
0923 17:28:09.718835 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.718839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.718904 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.719036 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:09.720711 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.721730 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.723883 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.724521 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.724652 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.726762 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:09.726767 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:09.726771 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:09.726805 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:09.729596 (+ 2791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.730382 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.730388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.730396 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.730591 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.731083 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.731089 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.731092 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.731512 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:09.731705 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.732171 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.732442 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:28:09.739374 9786 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:40833
I20260923 17:28:09.739392 9920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:40833 every 8 connection(s)
I20260923 17:28:09.740569 9786 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:09.742723 9921 heartbeater.cc:347] Connected to a master server at 127.8.29.254:46175
I20260923 17:28:09.742955 9921 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:09.743605 9921 heartbeater.cc:511] Master 127.8.29.254:46175 requested a full tablet report, sending...
I20260923 17:28:09.744318 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 9786
I20260923 17:28:09.744536 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:09.745126 8311 external_mini_cluster.cc:1428] Setting key f975355b4d992eab3f9374005da1e9c7
I20260923 17:28:09.745245 9435 ts_manager.cc:194] Registered new tserver with Master: 1136da8d8da14bc89cab48bdf49b9d19 (127.8.29.195:40833)
I20260923 17:28:09.746321 9435 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:36837
I20260923 17:28:09.748224 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:28:09 dist-test-slave-p7jt krb5kdc[9391](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:28:09 dist-test-slave-p7jt krb5kdc[9391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:09.793982 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.779310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41170 (local address 127.8.29.254:46175)
0923 17:28:09.779488 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.779493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.779803 (+ 310us) server_negotiation.cc:407] Connection header received
0923 17:28:09.779970 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.779974 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.780045 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.780138 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:09.781644 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.782400 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.784037 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.784512 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.784636 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.788571 (+ 3935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:09.788576 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:09.788579 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:09.788620 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:09.791380 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.792082 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.792086 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.792089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.792237 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:09.792680 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:09.792682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:09.792684 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:09.792955 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:09.793138 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.793455 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.793652 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:28:09.798641 9435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:09.805886 9435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:09.832247 9934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.819229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:42327 (local address 127.0.0.1:41074)
0923 17:28:09.823650 (+ 4421us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:09.823667 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:09.823774 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:09.824696 (+ 922us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:09.824700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:09.824726 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:09.825152 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.825166 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.826922 (+ 1756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.826927 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.829368 (+ 2441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.829381 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.831205 (+ 1824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.831211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.831401 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.831463 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 17:28:09.831597 (+ 134us) client_negotiation.cc:240] Negotiation successful
0923 17:28:09.831802 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4304,"mutex_wait_us":1,"thread_start_us":750,"threads_started":1}
I20260923 17:28:09.832365 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.819803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41074 (local address 127.8.29.194:42327)
0923 17:28:09.823356 (+ 3553us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.823365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.823913 (+ 548us) server_negotiation.cc:407] Connection header received
0923 17:28:09.824034 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.824042 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.824314 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.824496 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:09.825339 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.826762 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.829551 (+ 2789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.830979 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.831135 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.831347 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.831901 (+ 554us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.832089 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3409,"thread_start_us":3278,"threads_started":1}
I20260923 17:28:09.832878 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.818743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.8.29.193:41559)
0923 17:28:09.822979 (+ 4236us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.822990 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.823023 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:28:09.823138 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.823147 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.823441 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.823662 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:09.824396 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.827359 (+ 2963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.830273 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.831440 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.831578 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.831783 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.832366 (+ 583us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.832558 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4046,"thread_start_us":3892,"threads_started":1}
I20260923 17:28:09.833411 9936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.818646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:41559 (local address 127.0.0.1:38138)
0923 17:28:09.820000 (+ 1354us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:09.820049 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:09.820233 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:09.823698 (+ 3465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:09.823705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:09.823738 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:09.824137 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.824149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.827509 (+ 3360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.827515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.830086 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.830100 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.831678 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.831685 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.831867 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.831899 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:28:09.831985 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 17:28:09.832127 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1064,"mutex_wait_us":27,"thread_start_us":4765,"threads_started":1}
I20260923 17:28:09.836650 9712 tablet_service.cc:1467] Processing CreateTablet for tablet 5195567e095341eea95f4e4f905683e4 (DEFAULT_TABLE table=test-table [id=d599e48e8057450f9e2b01564f4dd915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:09.837245 9938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.824044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:40833 (local address 127.0.0.1:33396)
0923 17:28:09.828642 (+ 4598us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:09.828663 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:09.828790 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:09.830007 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:09.830012 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:09.830036 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:09.830401 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.830412 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.832364 (+ 1952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.832369 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.834356 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.834372 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.836471 (+ 2099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.836478 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.836667 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.836702 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:28:09.836798 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:28:09.836935 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4460,"thread_start_us":4293,"threads_started":1}
I20260923 17:28:09.837368 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.824090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33396 (local address 127.8.29.195:40833)
0923 17:28:09.828167 (+ 4077us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.828176 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.828786 (+ 610us) server_negotiation.cc:407] Connection header received
0923 17:28:09.829215 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.829225 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.829520 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.829723 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:09.830654 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.832210 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.834595 (+ 2385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.835991 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.836119 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.836342 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.836882 (+ 540us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.837057 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3887,"thread_start_us":3702,"threads_started":1}
I20260923 17:28:09.837033 9573 tablet_service.cc:1467] Processing CreateTablet for tablet 5195567e095341eea95f4e4f905683e4 (DEFAULT_TABLE table=test-table [id=d599e48e8057450f9e2b01564f4dd915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:09.838336 9712 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5195567e095341eea95f4e4f905683e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:09.838420 9573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5195567e095341eea95f4e4f905683e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:09.841426 9851 tablet_service.cc:1467] Processing CreateTablet for tablet 5195567e095341eea95f4e4f905683e4 (DEFAULT_TABLE table=test-table [id=d599e48e8057450f9e2b01564f4dd915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:09.843214 9851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5195567e095341eea95f4e4f905683e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:09.850576 9940 tablet_bootstrap.cc:492] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: Bootstrap starting.
I20260923 17:28:09.853210 9941 tablet_bootstrap.cc:492] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: Bootstrap starting.
I20260923 17:28:09.856026 9940 tablet_bootstrap.cc:654] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:09.856628 9941 tablet_bootstrap.cc:654] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:09.857357 9940 log.cc:824] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:09.857903 9941 log.cc:824] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:09.861001 9940 tablet_bootstrap.cc:492] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: No bootstrap required, opened a new log
I20260923 17:28:09.861343 9940 ts_tablet_manager.cc:1397] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260923 17:28:09.861325 9942 tablet_bootstrap.cc:492] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: Bootstrap starting.
I20260923 17:28:09.864249 9941 tablet_bootstrap.cc:492] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: No bootstrap required, opened a new log
I20260923 17:28:09.864567 9941 ts_tablet_manager.cc:1397] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260923 17:28:09.865304 9942 tablet_bootstrap.cc:654] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:09.866730 9942 log.cc:824] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:09.869817 9940 raft_consensus.cc:348] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } }
I20260923 17:28:09.870303 9940 raft_consensus.cc:374] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:09.870404 9940 raft_consensus.cc:729] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6378567c064c83a7431db623cade49, State: Initialized, Role: FOLLOWER
I20260923 17:28:09.870625 9942 tablet_bootstrap.cc:492] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: No bootstrap required, opened a new log
I20260923 17:28:09.870925 9942 ts_tablet_manager.cc:1397] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260923 17:28:09.871096 9940 consensus_queue.cc:260] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } }
I20260923 17:28:09.872947 9940 ts_tablet_manager.cc:1428] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260923 17:28:09.875095 9941 raft_consensus.cc:348] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } }
I20260923 17:28:09.875583 9941 raft_consensus.cc:374] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:09.875684 9941 raft_consensus.cc:729] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb1cd1625774ada9d54bf3fcc25df0b, State: Initialized, Role: FOLLOWER
I20260923 17:28:09.876312 9941 consensus_queue.cc:260] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } }
I20260923 17:28:09.877476 9782 heartbeater.cc:503] Master 127.8.29.254:46175 was elected leader, sending a full tablet report...
I20260923 17:28:09.878201 9941 ts_tablet_manager.cc:1428] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260923 17:28:09.880860 9942 raft_consensus.cc:348] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } }
I20260923 17:28:09.881348 9942 raft_consensus.cc:374] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:09.881458 9942 raft_consensus.cc:729] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1136da8d8da14bc89cab48bdf49b9d19, State: Initialized, Role: FOLLOWER
I20260923 17:28:09.882174 9942 consensus_queue.cc:260] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } }
I20260923 17:28:09.883415 9921 heartbeater.cc:503] Master 127.8.29.254:46175 was elected leader, sending a full tablet report...
I20260923 17:28:09.884114 9942 ts_tablet_manager.cc:1428] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: Time spent starting tablet: real 0.013s user 0.011s sys 0.002s
I20260923 17:28:09.949512 9948 raft_consensus.cc:482] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:09.949899 9948 raft_consensus.cc:504] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } }
I20260923 17:28:09.951640 9948 leader_election.cc:288] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f6378567c064c83a7431db623cade49 (127.8.29.193:41559), 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327)
W20260923 17:28:09.954218 9644 tablet.cc:2369] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:09.958294 9810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.951870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:41559 (local address 127.8.29.195:36389)
0923 17:28:09.952092 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:09.952112 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:09.952256 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:09.952897 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:09.952902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:09.952933 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:09.953133 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.953146 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.954317 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.954322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.956288 (+ 1966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.956303 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.957656 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.957660 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.957808 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.957835 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:28:09.957907 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 17:28:09.958004 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 17:28:09.958350 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.952203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:36389 (local address 127.8.29.193:41559)
0923 17:28:09.952547 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.952554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.952572 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:09.952673 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.952678 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.952765 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.952862 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:09.953317 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.954183 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.956460 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.957495 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.957659 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.957732 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.957967 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.958140 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172}
I20260923 17:28:09.958914 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.952559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:50411 (local address 127.8.29.194:42327)
0923 17:28:09.952748 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.952752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.952769 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:28:09.952898 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.952903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.952972 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.953115 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:09.953704 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.954704 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.956861 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.957977 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.958138 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.958205 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.958596 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.958712 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260923 17:28:09.959617 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.951958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:42327 (local address 127.8.29.195:50411)
0923 17:28:09.952656 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:09.952673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:09.952775 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:09.953142 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:09.953146 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:09.953168 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:09.953527 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.953539 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.954834 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.954842 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.956703 (+ 1861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:09.956713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.958200 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.958206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:09.958363 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.958396 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:28:09.959104 (+ 708us) client_negotiation.cc:240] Negotiation successful
0923 17:28:09.959289 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":344,"threads_started":1}
I20260923 17:28:09.959429 9593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5195567e095341eea95f4e4f905683e4" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6378567c064c83a7431db623cade49" is_pre_election: true
I20260923 17:28:09.960027 9593 raft_consensus.cc:2454] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 0.
I20260923 17:28:09.960637 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5195567e095341eea95f4e4f905683e4" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" is_pre_election: true
I20260923 17:28:09.960736 9804 leader_election.cc:302] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6378567c064c83a7431db623cade49, 1136da8d8da14bc89cab48bdf49b9d19; no voters:
I20260923 17:28:09.961141 9948 raft_consensus.cc:2781] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:09.961155 9737 raft_consensus.cc:2454] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 0.
I20260923 17:28:09.961319 9948 raft_consensus.cc:482] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:09.961431 9948 raft_consensus.cc:3037] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:09.963806 9948 raft_consensus.cc:504] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } }
I20260923 17:28:09.964404 9948 leader_election.cc:288] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 election: Requested vote from peers 0f6378567c064c83a7431db623cade49 (127.8.29.193:41559), 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327)
I20260923 17:28:09.964896 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5195567e095341eea95f4e4f905683e4" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b"
I20260923 17:28:09.964895 9593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5195567e095341eea95f4e4f905683e4" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6378567c064c83a7431db623cade49"
I20260923 17:28:09.965085 9737 raft_consensus.cc:3037] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:09.965116 9593 raft_consensus.cc:3037] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:09.967336 9593 raft_consensus.cc:2454] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 1.
I20260923 17:28:09.967357 9737 raft_consensus.cc:2454] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 1.
I20260923 17:28:09.967818 9804 leader_election.cc:302] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f6378567c064c83a7431db623cade49, 1136da8d8da14bc89cab48bdf49b9d19, 1fb1cd1625774ada9d54bf3fcc25df0b; no voters:
I20260923 17:28:09.968201 9948 raft_consensus.cc:2781] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:09.968614 9948 raft_consensus.cc:686] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 LEADER]: Becoming Leader. State: Replica: 1136da8d8da14bc89cab48bdf49b9d19, State: Running, Role: LEADER
I20260923 17:28:09.969139 9948 consensus_queue.cc:237] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } }
W20260923 17:28:09.974009 9783 tablet.cc:2369] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:09.973810 9435 catalog_manager.cc:5495] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1136da8d8da14bc89cab48bdf49b9d19 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } health_report { overall_health: UNKNOWN } } }
W20260923 17:28:09.992115 9922 tablet.cc:2369] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:09.996845 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:09.990687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.8.29.195:40833)
0923 17:28:09.990866 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:09.990870 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:09.990890 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:28:09.991182 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:09.991186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:09.991275 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:09.991403 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:09.991829 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.992923 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.994319 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:09.994814 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:09.994951 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:09.995558 (+ 607us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:09.996123 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:09.996243 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:09.996548 (+ 305us) server_negotiation.cc:299] Negotiation successful
0923 17:28:09.996631 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:28:10.005875 9737 raft_consensus.cc:1261] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Refusing update from remote peer 1136da8d8da14bc89cab48bdf49b9d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:10.005880 9593 raft_consensus.cc:1261] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Refusing update from remote peer 1136da8d8da14bc89cab48bdf49b9d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:10.007102 9950 consensus_queue.cc:1035] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:10.007582 9948 consensus_queue.cc:1035] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:10.041703 9435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:10.042102 9435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:10.046952 9435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c: Sending DeleteTablet for 3 replicas of tablet 5195567e095341eea95f4e4f905683e4
I20260923 17:28:10.048499 9851 tablet_service.cc:1514] Processing DeleteTablet for tablet 5195567e095341eea95f4e4f905683e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:10 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:33396
I20260923 17:28:10.048539 9712 tablet_service.cc:1514] Processing DeleteTablet for tablet 5195567e095341eea95f4e4f905683e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:10 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:41074
I20260923 17:28:10.048625 9573 tablet_service.cc:1514] Processing DeleteTablet for tablet 5195567e095341eea95f4e4f905683e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:10 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:38138
I20260923 17:28:10.049541 9963 tablet_replica.cc:330] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: stopping tablet replica
I20260923 17:28:10.050153 9965 tablet_replica.cc:330] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: stopping tablet replica
I20260923 17:28:10.050763 9965 raft_consensus.cc:2229] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:10.050993 9435 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:10.051759 9435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:10.051632 9963 raft_consensus.cc:2229] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:10.052410 9965 raft_consensus.cc:2258] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:10.053200 9963 raft_consensus.cc:2258] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:10.059399 9963 ts_tablet_manager.cc:1905] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:10.060534 9573 tablet_service.cc:1467] Processing CreateTablet for tablet a98b7c033f23416aa83d460f361d2b7c (DEFAULT_TABLE table=test-table [id=46204f7a46ae43299eb7efce1dbc091d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:10.061012 9573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98b7c033f23416aa83d460f361d2b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:10.061538 9851 tablet_service.cc:1467] Processing CreateTablet for tablet a98b7c033f23416aa83d460f361d2b7c (DEFAULT_TABLE table=test-table [id=46204f7a46ae43299eb7efce1dbc091d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:10.061996 9851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98b7c033f23416aa83d460f361d2b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:10.064059 9965 ts_tablet_manager.cc:1905] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:10.067032 9711 tablet_service.cc:1467] Processing CreateTablet for tablet a98b7c033f23416aa83d460f361d2b7c (DEFAULT_TABLE table=test-table [id=46204f7a46ae43299eb7efce1dbc091d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:10.067931 9711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a98b7c033f23416aa83d460f361d2b7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:10.070180 9964 tablet_replica.cc:330] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: stopping tablet replica
I20260923 17:28:10.070348 9942 tablet_bootstrap.cc:492] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: Bootstrap starting.
I20260923 17:28:10.070868 9964 raft_consensus.cc:2229] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:10.071458 9964 raft_consensus.cc:2258] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:10.072149 9942 tablet_bootstrap.cc:654] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:10.076217 9964 ts_tablet_manager.cc:1905] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:10.076299 9940 tablet_bootstrap.cc:492] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: Bootstrap starting.
I20260923 17:28:10.077813 9941 tablet_bootstrap.cc:492] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: Bootstrap starting.
I20260923 17:28:10.078001 9940 tablet_bootstrap.cc:654] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:10.079738 9941 tablet_bootstrap.cc:654] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:10.085752 9942 tablet_bootstrap.cc:492] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: No bootstrap required, opened a new log
I20260923 17:28:10.086133 9942 ts_tablet_manager.cc:1397] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.000s
I20260923 17:28:10.088012 9942 raft_consensus.cc:348] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.089380 9942 raft_consensus.cc:374] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:10.089665 9942 raft_consensus.cc:729] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1136da8d8da14bc89cab48bdf49b9d19, State: Initialized, Role: FOLLOWER
I20260923 17:28:10.090358 9942 consensus_queue.cc:260] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.090866 9940 tablet_bootstrap.cc:492] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: No bootstrap required, opened a new log
I20260923 17:28:10.091125 9940 ts_tablet_manager.cc:1397] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260923 17:28:10.091478 9963 ts_tablet_manager.cc:1918] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:10.092657 9940 raft_consensus.cc:348] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.093010 9965 ts_tablet_manager.cc:1918] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:10.093416 9965 log.cc:1196] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/5195567e095341eea95f4e4f905683e4
I20260923 17:28:10.094120 9965 ts_tablet_manager.cc:1939] T 5195567e095341eea95f4e4f905683e4 P 0f6378567c064c83a7431db623cade49: Deleting consensus metadata
I20260923 17:28:10.091763 9963 log.cc:1196] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/5195567e095341eea95f4e4f905683e4
I20260923 17:28:10.095050 9963 ts_tablet_manager.cc:1939] T 5195567e095341eea95f4e4f905683e4 P 1136da8d8da14bc89cab48bdf49b9d19: Deleting consensus metadata
I20260923 17:28:10.095319 9942 ts_tablet_manager.cc:1428] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: Time spent starting tablet: real 0.009s user 0.001s sys 0.003s
I20260923 17:28:10.097008 9941 tablet_bootstrap.cc:492] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: No bootstrap required, opened a new log
I20260923 17:28:10.097266 9941 ts_tablet_manager.cc:1397] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20260923 17:28:10.093015 9940 raft_consensus.cc:374] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:10.097831 9940 raft_consensus.cc:729] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6378567c064c83a7431db623cade49, State: Initialized, Role: FOLLOWER
I20260923 17:28:10.098047 9422 catalog_manager.cc:4841] TS 1136da8d8da14bc89cab48bdf49b9d19 (127.8.29.195:40833): tablet 5195567e095341eea95f4e4f905683e4 (table test-table [id=d599e48e8057450f9e2b01564f4dd915]) successfully deleted
I20260923 17:28:10.098217 9941 raft_consensus.cc:348] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.098203 9940 consensus_queue.cc:260] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.098436 9941 raft_consensus.cc:374] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:10.098580 9941 raft_consensus.cc:729] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb1cd1625774ada9d54bf3fcc25df0b, State: Initialized, Role: FOLLOWER
I20260923 17:28:10.098835 9941 consensus_queue.cc:260] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.098964 9940 ts_tablet_manager.cc:1428] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260923 17:28:10.099310 9941 ts_tablet_manager.cc:1428] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 17:28:10.100539 9964 ts_tablet_manager.cc:1918] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:10.100695 9964 log.cc:1196] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/5195567e095341eea95f4e4f905683e4
I20260923 17:28:10.101223 9964 ts_tablet_manager.cc:1939] T 5195567e095341eea95f4e4f905683e4 P 1fb1cd1625774ada9d54bf3fcc25df0b: Deleting consensus metadata
I20260923 17:28:10.101733 9422 catalog_manager.cc:4841] TS 0f6378567c064c83a7431db623cade49 (127.8.29.193:41559): tablet 5195567e095341eea95f4e4f905683e4 (table test-table [id=d599e48e8057450f9e2b01564f4dd915]) successfully deleted
I20260923 17:28:10.103029 9420 catalog_manager.cc:4841] TS 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327): tablet 5195567e095341eea95f4e4f905683e4 (table test-table [id=d599e48e8057450f9e2b01564f4dd915]) successfully deleted
I20260923 17:28:10.333932 9950 raft_consensus.cc:482] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:10.334190 9950 raft_consensus.cc:504] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.334893 9950 leader_election.cc:288] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f6378567c064c83a7431db623cade49 (127.8.29.193:41559), 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327)
I20260923 17:28:10.335575 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98b7c033f23416aa83d460f361d2b7c" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" is_pre_election: true
I20260923 17:28:10.335578 9593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98b7c033f23416aa83d460f361d2b7c" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6378567c064c83a7431db623cade49" is_pre_election: true
I20260923 17:28:10.335825 9737 raft_consensus.cc:2454] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 0.
I20260923 17:28:10.335878 9593 raft_consensus.cc:2454] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 0.
I20260923 17:28:10.336177 9946 raft_consensus.cc:482] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:10.336342 9802 leader_election.cc:302] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1136da8d8da14bc89cab48bdf49b9d19, 1fb1cd1625774ada9d54bf3fcc25df0b; no voters:
I20260923 17:28:10.336376 9946 raft_consensus.cc:504] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.336860 9950 raft_consensus.cc:2781] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:10.336977 9950 raft_consensus.cc:482] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:10.337105 9950 raft_consensus.cc:3037] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:10.338090 9946 leader_election.cc:288] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327), 1136da8d8da14bc89cab48bdf49b9d19 (127.8.29.195:40833)
I20260923 17:28:10.340392 9950 raft_consensus.cc:504] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.342438 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98b7c033f23416aa83d460f361d2b7c" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b"
I20260923 17:28:10.342732 9737 raft_consensus.cc:3037] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:10.345847 9737 raft_consensus.cc:2454] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 1.
I20260923 17:28:10.346654 9593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98b7c033f23416aa83d460f361d2b7c" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6378567c064c83a7431db623cade49"
I20260923 17:28:10.346921 9593 raft_consensus.cc:3037] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:10.350768 9802 leader_election.cc:302] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1136da8d8da14bc89cab48bdf49b9d19, 1fb1cd1625774ada9d54bf3fcc25df0b; no voters:
I20260923 17:28:10.351181 9959 raft_consensus.cc:2781] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:10.351456 9959 raft_consensus.cc:686] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 LEADER]: Becoming Leader. State: Replica: 1136da8d8da14bc89cab48bdf49b9d19, State: Running, Role: LEADER
I20260923 17:28:10.350261 9593 raft_consensus.cc:2454] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 1.
I20260923 17:28:10.351864 9959 consensus_queue.cc:237] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.353809 9950 leader_election.cc:288] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 election: Requested vote from peers 0f6378567c064c83a7431db623cade49 (127.8.29.193:41559), 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327)
I20260923 17:28:10.357551 9937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.338720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:57731 (local address 127.8.29.194:42327)
0923 17:28:10.338982 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:10.338989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:10.349913 (+ 10924us) server_negotiation.cc:407] Connection header received
0923 17:28:10.350511 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:10.350517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:10.350610 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:10.350679 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:10.351425 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.352452 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.355142 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.356335 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.356471 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.356544 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:10.357079 (+ 535us) server_negotiation.cc:299] Negotiation successful
0923 17:28:10.357241 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260923 17:28:10.358321 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.338409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:42327 (local address 127.8.29.193:57731)
0923 17:28:10.349751 (+ 11342us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:10.349780 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:10.350283 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:10.350859 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:10.350864 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:10.350899 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:10.351220 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.351230 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.352708 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.352713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.354946 (+ 2233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.354959 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.356740 (+ 1781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.356747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.356900 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.356935 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:28:10.357813 (+ 878us) client_negotiation.cc:240] Negotiation successful
0923 17:28:10.357967 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11134,"thread_start_us":12065,"threads_started":1}
I20260923 17:28:10.358994 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98b7c033f23416aa83d460f361d2b7c" candidate_uuid: "0f6378567c064c83a7431db623cade49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" is_pre_election: true
I20260923 17:28:10.359370 9737 raft_consensus.cc:2379] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0f6378567c064c83a7431db623cade49 in current term 1: Already voted for candidate 1136da8d8da14bc89cab48bdf49b9d19 in this term.
I20260923 17:28:10.360036 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.341392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:52269 (local address 127.8.29.195:40833)
0923 17:28:10.341711 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:10.341718 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:10.350407 (+ 8689us) server_negotiation.cc:407] Connection header received
0923 17:28:10.351525 (+ 1118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:10.351530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:10.351629 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:10.351713 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:10.353171 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.354423 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.356887 (+ 2464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.358304 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.358435 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.358543 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:10.359617 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0923 17:28:10.359760 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260923 17:28:10.360713 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.338705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:40833 (local address 127.8.29.193:52269)
0923 17:28:10.350304 (+ 11599us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:10.350321 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:10.350420 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:10.352497 (+ 2077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:10.352502 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:10.352532 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:10.352979 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.352991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.354709 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.354714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.356704 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.356718 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.359038 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.359044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.359352 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.359388 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:28:10.360317 (+ 929us) client_negotiation.cc:240] Negotiation successful
0923 17:28:10.360435 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11480,"mutex_wait_us":1,"thread_start_us":11238,"threads_started":1}
I20260923 17:28:10.361594 9875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a98b7c033f23416aa83d460f361d2b7c" candidate_uuid: "0f6378567c064c83a7431db623cade49" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1136da8d8da14bc89cab48bdf49b9d19" is_pre_election: true
I20260923 17:28:10.362965 9525 leader_election.cc:302] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0f6378567c064c83a7431db623cade49; no voters: 1136da8d8da14bc89cab48bdf49b9d19, 1fb1cd1625774ada9d54bf3fcc25df0b
I20260923 17:28:10.363344 9946 raft_consensus.cc:2726] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 17:28:10.363865 9435 catalog_manager.cc:5495] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1136da8d8da14bc89cab48bdf49b9d19 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } health_report { overall_health: HEALTHY } } }
Sep 23 17:28:10 dist-test-slave-p7jt krb5kdc[9391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184487, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:10.428735 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.412839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46175 (local address 127.0.0.1:41178)
0923 17:28:10.413694 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:10.413715 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:10.413847 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:10.414265 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:10.414269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:10.414990 (+ 721us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:10.415192 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.415201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.416767 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.416771 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.417741 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.417752 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.418818 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.418824 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.418972 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.419764 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:10.419787 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:10.421971 (+ 2184us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:10.426090 (+ 4119us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:10.426104 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:10.426111 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:10.426622 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:10.427072 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:10.427076 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:10.427078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:10.427237 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:10.427909 (+ 672us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:10.427916 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:10.428089 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 17:28:10.428218 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 17:28:10.428424 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":564,"threads_started":1}
I20260923 17:28:10.429169 9977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.412855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41178 (local address 127.8.29.254:46175)
0923 17:28:10.413580 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:10.413588 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:10.413813 (+ 225us) server_negotiation.cc:407] Connection header received
0923 17:28:10.413966 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:10.413973 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:10.414060 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:10.414205 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:10.415371 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.416594 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.417906 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.418634 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.418785 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.422214 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:10.422223 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:10.422229 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:10.422273 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:10.425908 (+ 3635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:10.426756 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:10.426762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:10.426766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:10.426939 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:10.427361 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:10.427367 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:10.427371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:10.427778 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:10.427946 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:10.428397 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 17:28:10.428640 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":337,"threads_started":1}
I20260923 17:28:10.434883 9435 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:41178:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:28:10.436302 9435 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:10.444481 9573 tablet_service.cc:1467] Processing CreateTablet for tablet 9aba23bc960647f4a65c354933e6d203 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a60b6f6327c4447c827315dbcd11bc4e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:28:10.444720 9851 tablet_service.cc:1467] Processing CreateTablet for tablet 9aba23bc960647f4a65c354933e6d203 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a60b6f6327c4447c827315dbcd11bc4e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:28:10.445067 9573 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aba23bc960647f4a65c354933e6d203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:10.445422 9851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aba23bc960647f4a65c354933e6d203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:10.451356 9940 tablet_bootstrap.cc:492] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: Bootstrap starting.
I20260923 17:28:10.451315 9711 tablet_service.cc:1467] Processing CreateTablet for tablet 9aba23bc960647f4a65c354933e6d203 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=a60b6f6327c4447c827315dbcd11bc4e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:28:10.452054 9711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9aba23bc960647f4a65c354933e6d203. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:10.453941 9940 tablet_bootstrap.cc:654] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:10.459870 9942 tablet_bootstrap.cc:492] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: Bootstrap starting.
I20260923 17:28:10.461905 9942 tablet_bootstrap.cc:654] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:10.462347 9940 tablet_bootstrap.cc:492] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: No bootstrap required, opened a new log
I20260923 17:28:10.462653 9940 ts_tablet_manager.cc:1397] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.008s
I20260923 17:28:10.462970 9941 tablet_bootstrap.cc:492] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: Bootstrap starting.
I20260923 17:28:10.463577 9940 raft_consensus.cc:348] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.463837 9940 raft_consensus.cc:374] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:10.463915 9940 raft_consensus.cc:729] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f6378567c064c83a7431db623cade49, State: Initialized, Role: FOLLOWER
I20260923 17:28:10.464299 9940 consensus_queue.cc:260] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.464862 9940 ts_tablet_manager.cc:1428] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 17:28:10.464807 9946 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.465026 9941 tablet_bootstrap.cc:654] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:10.465124 9946 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:10.467343 9942 tablet_bootstrap.cc:492] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: No bootstrap required, opened a new log
I20260923 17:28:10.467604 9942 ts_tablet_manager.cc:1397] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20260923 17:28:10.468484 9942 raft_consensus.cc:348] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.468782 9942 raft_consensus.cc:374] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:10.468868 9942 raft_consensus.cc:729] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1136da8d8da14bc89cab48bdf49b9d19, State: Initialized, Role: FOLLOWER
I20260923 17:28:10.469193 9942 consensus_queue.cc:260] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.469866 9942 ts_tablet_manager.cc:1428] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:28:10.469740 9959 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.470430 9959 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:10.471853 9941 tablet_bootstrap.cc:492] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: No bootstrap required, opened a new log
I20260923 17:28:10.472033 9941 ts_tablet_manager.cc:1397] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20260923 17:28:10.472806 9941 raft_consensus.cc:348] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.473012 9941 raft_consensus.cc:374] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:10.473090 9941 raft_consensus.cc:729] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fb1cd1625774ada9d54bf3fcc25df0b, State: Initialized, Role: FOLLOWER
I20260923 17:28:10.473333 9941 consensus_queue.cc:260] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.473901 9941 ts_tablet_manager.cc:1428] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260923 17:28:10.473771 9947 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.474149 9947 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:10.669330 9959 raft_consensus.cc:482] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:10.669543 9959 raft_consensus.cc:504] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.670215 9959 leader_election.cc:288] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0f6378567c064c83a7431db623cade49 (127.8.29.193:41559), 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327)
I20260923 17:28:10.670934 9593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aba23bc960647f4a65c354933e6d203" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6378567c064c83a7431db623cade49" is_pre_election: true
I20260923 17:28:10.670943 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aba23bc960647f4a65c354933e6d203" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" is_pre_election: true
I20260923 17:28:10.671221 9593 raft_consensus.cc:2454] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 0.
I20260923 17:28:10.671221 9737 raft_consensus.cc:2454] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 0.
I20260923 17:28:10.671698 9804 leader_election.cc:302] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0f6378567c064c83a7431db623cade49, 1136da8d8da14bc89cab48bdf49b9d19, 1fb1cd1625774ada9d54bf3fcc25df0b; no voters:
I20260923 17:28:10.672006 9959 raft_consensus.cc:2781] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:10.672098 9959 raft_consensus.cc:482] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:10.672202 9959 raft_consensus.cc:3037] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:10.674127 9959 raft_consensus.cc:504] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.674719 9959 leader_election.cc:288] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 election: Requested vote from peers 0f6378567c064c83a7431db623cade49 (127.8.29.193:41559), 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327)
I20260923 17:28:10.675215 9593 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aba23bc960647f4a65c354933e6d203" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f6378567c064c83a7431db623cade49"
I20260923 17:28:10.675246 9737 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9aba23bc960647f4a65c354933e6d203" candidate_uuid: "1136da8d8da14bc89cab48bdf49b9d19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b"
I20260923 17:28:10.675433 9593 raft_consensus.cc:3037] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:10.675447 9737 raft_consensus.cc:3037] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:10.677302 9593 raft_consensus.cc:2454] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 1.
I20260923 17:28:10.677529 9737 raft_consensus.cc:2454] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1136da8d8da14bc89cab48bdf49b9d19 in term 1.
I20260923 17:28:10.677721 9804 leader_election.cc:302] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f6378567c064c83a7431db623cade49, 1136da8d8da14bc89cab48bdf49b9d19; no voters:
I20260923 17:28:10.677973 9959 raft_consensus.cc:2781] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:10.678126 9959 raft_consensus.cc:686] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 LEADER]: Becoming Leader. State: Replica: 1136da8d8da14bc89cab48bdf49b9d19, State: Running, Role: LEADER
I20260923 17:28:10.678398 9959 consensus_queue.cc:237] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } }
I20260923 17:28:10.679921 9950 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: TxnStatusTablet state changed. Reason: New leader 1136da8d8da14bc89cab48bdf49b9d19. Latest consensus state: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.680194 9950 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:28:10.680680 9983 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:28:10.680871 9983 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:28:10.681375 9436 catalog_manager.cc:5495] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1136da8d8da14bc89cab48bdf49b9d19 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } health_report { overall_health: HEALTHY } } }
I20260923 17:28:10.757362 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.750835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.8.29.195:40833)
0923 17:28:10.751017 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:10.751023 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:10.751043 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:28:10.751356 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:10.751361 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:10.751476 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:10.751575 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:10.751987 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.753104 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.755114 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.755841 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.755968 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.756549 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:10.756711 (+ 162us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:10.756799 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:10.756967 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 17:28:10.757061 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:28:10.757426 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.750735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:40833 (local address 127.0.0.1:33416)
0923 17:28:10.750960 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:10.750981 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:10.751141 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:10.751629 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:10.751634 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:10.751648 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:28:10.751841 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.751852 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.753264 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.753270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.754915 (+ 1645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.754932 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.756027 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.756034 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.756244 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.756329 (+ 85us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:28:10.756822 (+ 493us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:28:10.756844 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:28:10.756928 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:28:10.757110 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:28:10.765805 9737 raft_consensus.cc:1261] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Refusing update from remote peer 1136da8d8da14bc89cab48bdf49b9d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:10.766155 9593 raft_consensus.cc:1261] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Refusing update from remote peer 1136da8d8da14bc89cab48bdf49b9d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:10.767084 9959 consensus_queue.cc:1035] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:10.767702 9950 consensus_queue.cc:1035] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:10.783138 9946 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: TxnStatusTablet state changed. Reason: New leader 1136da8d8da14bc89cab48bdf49b9d19. Latest consensus state: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.783592 9946 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:10.783601 9947 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: TxnStatusTablet state changed. Reason: New leader 1136da8d8da14bc89cab48bdf49b9d19. Latest consensus state: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.784108 9947 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:10.787082 9986 mvcc.cc:205] Tried to move back new op lower bound from 7332595674167853056 to 7332595673821208576. Current Snapshot: MvccSnapshot[applied={T|T < 7332595674167853056}]
I20260923 17:28:10.788086 9948 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.788399 9948 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:28:10.792059 9959 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.792379 9959 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 1136da8d8da14bc89cab48bdf49b9d19: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:28:10.793704 9987 mvcc.cc:205] Tried to move back new op lower bound from 7332595674167853056 to 7332595673821208576. Current Snapshot: MvccSnapshot[applied={T|T < 7332595674167853056}]
I20260923 17:28:10.798238 9946 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.798528 9946 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 0f6378567c064c83a7431db623cade49: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:10.799082 9947 tablet_replica.cc:439] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1136da8d8da14bc89cab48bdf49b9d19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 } } peers { permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 } } peers { permanent_uuid: "1136da8d8da14bc89cab48bdf49b9d19" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40833 } } }
I20260923 17:28:10.799319 9947 tablet_replica.cc:442] T 9aba23bc960647f4a65c354933e6d203 P 1fb1cd1625774ada9d54bf3fcc25df0b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:10.813290 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.807671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:39079 (local address 127.8.29.195:40833)
0923 17:28:10.807864 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:10.807871 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:10.808318 (+ 447us) server_negotiation.cc:407] Connection header received
0923 17:28:10.808508 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:10.808513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:10.808597 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:10.808673 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:10.809167 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.809958 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.811472 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.811914 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.812021 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.812461 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:10.812596 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:10.812691 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:10.812835 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 17:28:10.812905 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260923 17:28:10.813344 9996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.807580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:40833 (local address 127.8.29.195:39079)
0923 17:28:10.808209 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:10.808224 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:10.808383 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:10.808705 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:10.808709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:10.808720 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:28:10.809025 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.809034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.810097 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.810103 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.811318 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:10.811333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.812101 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.812106 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:10.812225 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.812290 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:28:10.812714 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:28:10.812725 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:28:10.812799 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:28:10.812939 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":317,"threads_started":1}
I20260923 17:28:10.823593 9736 raft_consensus.cc:1261] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Refusing update from remote peer 1136da8d8da14bc89cab48bdf49b9d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:10.824139 9593 raft_consensus.cc:1261] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Refusing update from remote peer 1136da8d8da14bc89cab48bdf49b9d19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:10.824163 9950 consensus_queue.cc:1035] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fb1cd1625774ada9d54bf3fcc25df0b" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:10.824669 9950 consensus_queue.cc:1035] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f6378567c064c83a7431db623cade49" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:10.887324 9435 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:10.887549 9435 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:10.890553 9435 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a52c9f63e784608b51fba38e62b756c: Sending DeleteTablet for 3 replicas of tablet a98b7c033f23416aa83d460f361d2b7c
I20260923 17:28:10.891566 9573 tablet_service.cc:1514] Processing DeleteTablet for tablet a98b7c033f23416aa83d460f361d2b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:10 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:38138
I20260923 17:28:10.891566 9711 tablet_service.cc:1514] Processing DeleteTablet for tablet a98b7c033f23416aa83d460f361d2b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:10 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:41074
I20260923 17:28:10.891911 9850 tablet_service.cc:1514] Processing DeleteTablet for tablet a98b7c033f23416aa83d460f361d2b7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:10 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:33396
I20260923 17:28:10.899928 10002 tablet_replica.cc:330] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: stopping tablet replica
I20260923 17:28:10.900749 10002 raft_consensus.cc:2229] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:10.901321 10002 raft_consensus.cc:2258] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:10.902284 10003 tablet_replica.cc:330] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: stopping tablet replica
I20260923 17:28:10.903332 10002 ts_tablet_manager.cc:1905] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:10.903692 10003 raft_consensus.cc:2229] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:10.904206 10003 raft_consensus.cc:2258] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:10.906862 10003 ts_tablet_manager.cc:1905] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:10.916318 10003 ts_tablet_manager.cc:1918] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:28:10.920440 10001 tablet_replica.cc:330] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: stopping tablet replica
I20260923 17:28:10.920665 10001 raft_consensus.cc:2229] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:10.920905 10001 raft_consensus.cc:2258] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:10.921324 10002 ts_tablet_manager.cc:1918] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:28:10.921461 10002 log.cc:1196] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/a98b7c033f23416aa83d460f361d2b7c
I20260923 17:28:10.921897 10002 ts_tablet_manager.cc:1939] T a98b7c033f23416aa83d460f361d2b7c P 1fb1cd1625774ada9d54bf3fcc25df0b: Deleting consensus metadata
I20260923 17:28:10.923080 9420 catalog_manager.cc:4841] TS 1fb1cd1625774ada9d54bf3fcc25df0b (127.8.29.194:42327): tablet a98b7c033f23416aa83d460f361d2b7c (table test-table [id=46204f7a46ae43299eb7efce1dbc091d]) successfully deleted
I20260923 17:28:10.923398 10003 log.cc:1196] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/a98b7c033f23416aa83d460f361d2b7c
I20260923 17:28:10.923923 10003 ts_tablet_manager.cc:1939] T a98b7c033f23416aa83d460f361d2b7c P 1136da8d8da14bc89cab48bdf49b9d19: Deleting consensus metadata
I20260923 17:28:10.924121 10001 ts_tablet_manager.cc:1905] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:10.930610 10001 ts_tablet_manager.cc:1918] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:28:10.930866 10001 log.cc:1196] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/a98b7c033f23416aa83d460f361d2b7c
I20260923 17:28:10.931380 10001 ts_tablet_manager.cc:1939] T a98b7c033f23416aa83d460f361d2b7c P 0f6378567c064c83a7431db623cade49: Deleting consensus metadata
Sep 23 17:28:10 dist-test-slave-p7jt krb5kdc[9391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:10.931914 9422 catalog_manager.cc:4841] TS 1136da8d8da14bc89cab48bdf49b9d19 (127.8.29.195:40833): tablet a98b7c033f23416aa83d460f361d2b7c (table test-table [id=46204f7a46ae43299eb7efce1dbc091d]) successfully deleted
I20260923 17:28:10.932778 9422 catalog_manager.cc:4841] TS 0f6378567c064c83a7431db623cade49 (127.8.29.193:41559): tablet a98b7c033f23416aa83d460f361d2b7c (table test-table [id=46204f7a46ae43299eb7efce1dbc091d]) successfully deleted
I20260923 17:28:10.940261 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.905242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.8.29.193:41559)
0923 17:28:10.919030 (+ 13788us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:10.919038 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:10.921092 (+ 2054us) server_negotiation.cc:407] Connection header received
0923 17:28:10.921170 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:10.921175 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:10.921267 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:10.921398 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:10.923216 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.924293 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.926328 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.927109 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.927238 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.933106 (+ 5868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:10.933126 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:10.933147 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:10.933193 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:10.936986 (+ 3793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:10.937858 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:10.937869 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:10.937872 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:10.938034 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:10.938382 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:10.938386 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:10.938389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:10.938868 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:10.939165 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:10.939634 (+ 469us) server_negotiation.cc:299] Negotiation successful
0923 17:28:10.939892 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13604,"thread_start_us":14764,"threads_started":1}
W20260923 17:28:10.941071 9638 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38148
I20260923 17:28:10.959342 10009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.944387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41190 (local address 127.8.29.254:46175)
0923 17:28:10.944947 (+ 560us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:10.944953 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:10.944969 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:28:10.945021 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:10.945025 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:10.945091 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:10.945181 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:10.946523 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.947278 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.948516 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.948978 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.949095 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.951875 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:10.951882 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:10.951885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:10.951952 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:10.955582 (+ 3630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:10.956603 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:10.956608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:10.956610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:10.956748 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:10.957286 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:10.957290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:10.957292 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:10.957662 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:10.958060 (+ 398us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:10.958558 (+ 498us) server_negotiation.cc:299] Negotiation successful
0923 17:28:10.958829 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":319,"threads_started":1}
W20260923 17:28:10.960256 9435 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41190
I20260923 17:28:10.963488 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 9507
I20260923 17:28:10.980275 10005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.964519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.8.29.193:41559)
0923 17:28:10.964739 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:10.964745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:10.964949 (+ 204us) server_negotiation.cc:407] Connection header received
0923 17:28:10.965089 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:10.965095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:10.965188 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:10.965294 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:10.966652 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.967763 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.969087 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:10.970126 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:10.970282 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:10.973717 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:10.973726 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:10.973731 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:10.973785 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:10.977326 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:10.978165 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:10.978171 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:10.978175 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:10.978348 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:10.978762 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:10.978767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:10.978770 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:10.979122 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:10.979377 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:10.979687 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 17:28:10.979937 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
W20260923 17:28:10.981031 9638 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38154
W20260923 17:28:10.981724 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 9507: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:28:10.981943 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 9507
I20260923 17:28:10.996899 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 9647
Sep 23 17:28:11 dist-test-slave-p7jt krb5kdc[9391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:28:11.016316 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:10.997858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41088 (local address 127.8.29.194:42327)
0923 17:28:10.998359 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:10.998366 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:10.998388 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:28:10.998476 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:10.998482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:10.998576 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:10.998691 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:11.000629 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:11.001664 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:11.003380 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:11.004059 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:11.004196 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:11.008991 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:11.009013 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:11.009035 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:11.009117 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:11.013139 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:11.013971 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:11.013983 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:11.013986 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:11.014170 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:11.014564 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:11.014570 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:11.014573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:11.015069 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:11.015383 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:11.015669 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 17:28:11.015930 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":235,"threads_started":1}
W20260923 17:28:11.017107 9777 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41088
W20260923 17:28:11.017851 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 9647: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:28:11.018023 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 9647
I20260923 17:28:11.032249 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 9786
Sep 23 17:28:11 dist-test-slave-p7jt krb5kdc[9391](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184489, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:28:11.048802 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:11.033106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33426 (local address 127.8.29.195:40833)
0923 17:28:11.033292 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:11.033298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:11.033318 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:28:11.033551 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:11.033556 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:11.033626 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:11.033720 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:11.035016 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:11.035816 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:11.037125 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:11.037601 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:11.037722 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:11.042195 (+ 4473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:11.042213 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:11.042234 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:11.042275 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:11.045536 (+ 3261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:11.046387 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:11.046396 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:11.046398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:11.046616 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:11.047141 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:11.047146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:11.047148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:11.047566 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:11.047882 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:11.048198 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 17:28:11.048467 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:28:11.049511 9916 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33426
W20260923 17:28:11.050282 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 9786: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:28:11.050431 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 9786
I20260923 17:28:11.078519 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 9404
I20260923 17:28:11.079761 9476 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:11.195621 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 9404
2026-09-23T17:28:11Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4748 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:11 dist-test-slave-p7jt krb5kdc[10015](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:11 dist-test-slave-p7jt krb5kdc[10015](info): set up 2 sockets
Sep 23 17:28:11 dist-test-slave-p7jt krb5kdc[10015](info): commencing operation
krb5kdc: starting...
W20260923 17:28:12.202499 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.968s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:12 dist-test-slave-p7jt krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:28:12.387117 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:42027
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:42027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:12.703748 10028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:12.704041 10028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:12.704169 10028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:12.711365 10028 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:12.711480 10028 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:28:12.711532 10028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:12.711617 10028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:12.711679 10028 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:12.711760 10028 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:12.722298 10028 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:42027
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:42027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:12.723631 10028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:12.725009 10028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:12.737062 10033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:12.737062 10036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:12.737115 10034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:12.738091 10028 server_base.cc:1034] running on GCE node
I20260923 17:28:12.738933 10028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:12.740442 10028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:12.741683 10028 hybrid_clock.cc:648] HybridClock initialized: now 1790184492741603 us; error 50 us; skew 500 ppm
I20260923 17:28:12.742038 10028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:12 dist-test-slave-p7jt krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184492, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:12.745882 10028 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:12.747442 10028 webserver.cc:460] Webserver started at http://127.8.29.254:35877/ using document root <none> and password file <none>
I20260923 17:28:12.748162 10028 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:12.748260 10028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:12.748570 10028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:12.751732 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "abd24a37b98c412dbe7304c850fe199a"
format_stamp: "Formatted at 2026-09-23 17:28:12 on dist-test-slave-p7jt"
server_key: "5947134c90fffea264149ad3df0274c8"
server_key_iv: "848a5f8a41a67e5e24fc8a4e71b577dc"
server_key_version: "encryptionkey@0"
I20260923 17:28:12.752548 10028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "abd24a37b98c412dbe7304c850fe199a"
format_stamp: "Formatted at 2026-09-23 17:28:12 on dist-test-slave-p7jt"
server_key: "5947134c90fffea264149ad3df0274c8"
server_key_iv: "848a5f8a41a67e5e24fc8a4e71b577dc"
server_key_version: "encryptionkey@0"
I20260923 17:28:12.757948 10028 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:28:12.761739 10043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:12.763445 10028 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:28:12.763643 10028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "abd24a37b98c412dbe7304c850fe199a"
format_stamp: "Formatted at 2026-09-23 17:28:12 on dist-test-slave-p7jt"
server_key: "5947134c90fffea264149ad3df0274c8"
server_key_iv: "848a5f8a41a67e5e24fc8a4e71b577dc"
server_key_version: "encryptionkey@0"
I20260923 17:28:12.763870 10028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:12.777544 10028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:12.782006 10028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:12.782280 10028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:12.802946 10028 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:42027
I20260923 17:28:12.802966 10104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:42027 every 8 connection(s)
I20260923 17:28:12.804427 10028 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:12.808414 10105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:12.813917 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 10028
I20260923 17:28:12.814129 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:12.814918 8311 external_mini_cluster.cc:1428] Setting key 736d3966bad5d4884e3eb0f9f5285ee2
I20260923 17:28:12.820412 10105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a: Bootstrap starting.
I20260923 17:28:12.824199 10105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:12.825536 10105 log.cc:824] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:12.829139 10105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a: No bootstrap required, opened a new log
Sep 23 17:28:12 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184492, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:12.837121 10105 raft_consensus.cc:348] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd24a37b98c412dbe7304c850fe199a" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 42027 } }
I20260923 17:28:12.837508 10105 raft_consensus.cc:374] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:12.837605 10105 raft_consensus.cc:729] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abd24a37b98c412dbe7304c850fe199a, State: Initialized, Role: FOLLOWER
I20260923 17:28:12.838310 10105 consensus_queue.cc:260] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd24a37b98c412dbe7304c850fe199a" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 42027 } }
I20260923 17:28:12.838616 10105 raft_consensus.cc:388] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:12.838757 10105 raft_consensus.cc:482] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:12.838939 10105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:12.839936 10108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:12.817305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.8.29.254:42027)
0923 17:28:12.818252 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:12.818271 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:12.818319 (+ 48us) server_negotiation.cc:407] Connection header received
0923 17:28:12.820287 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:12.820313 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:12.820812 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:12.821200 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:12.823056 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:12.824445 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:12.826255 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:12.827314 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:12.827464 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:12.832086 (+ 4622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:12.832097 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:12.832115 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:12.832160 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:12.835290 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:12.836024 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:12.836031 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:12.836039 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:12.836223 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:12.836605 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:12.836609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:12.836611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:12.837177 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:12.837452 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:12.837939 (+ 487us) server_negotiation.cc:299] Negotiation successful
0923 17:28:12.838404 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":456,"threads_started":1}
I20260923 17:28:12.841665 10105 raft_consensus.cc:504] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd24a37b98c412dbe7304c850fe199a" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 42027 } }
I20260923 17:28:12.842199 10105 leader_election.cc:302] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abd24a37b98c412dbe7304c850fe199a; no voters:
I20260923 17:28:12.843137 10105 leader_election.cc:288] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:12.843251 10110 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:12.843946 10110 raft_consensus.cc:686] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [term 1 LEADER]: Becoming Leader. State: Replica: abd24a37b98c412dbe7304c850fe199a, State: Running, Role: LEADER
I20260923 17:28:12.844604 10110 consensus_queue.cc:237] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd24a37b98c412dbe7304c850fe199a" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 42027 } }
I20260923 17:28:12.845690 10105 sys_catalog.cc:564] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:12.847921 10112 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [sys.catalog]: SysCatalogTable state changed. Reason: New leader abd24a37b98c412dbe7304c850fe199a. Latest consensus state: current_term: 1 leader_uuid: "abd24a37b98c412dbe7304c850fe199a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd24a37b98c412dbe7304c850fe199a" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 42027 } } }
I20260923 17:28:12.848429 10112 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:12.848954 10111 sys_catalog.cc:455] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abd24a37b98c412dbe7304c850fe199a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abd24a37b98c412dbe7304c850fe199a" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 42027 } } }
I20260923 17:28:12.849256 10111 sys_catalog.cc:458] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:12.855233 10115 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:12.860430 10115 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:12.869604 10115 catalog_manager.cc:1347] Generated new cluster ID: a63b00242e2e472183f8c03c7ca38196
I20260923 17:28:12.869733 10115 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:12.900094 10115 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:12.901432 10115 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:12.909976 10115 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a: Generated new TSK 0
I20260923 17:28:12.910914 10115 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
W20260923 17:28:13.942319 8311 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.8.29.193: real 1.021s user 0.001s sys 0.008s
I20260923 17:28:13.943315 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:42027
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:14.281857 10131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:14.282173 10131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:14.282308 10131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:14.290076 10131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:14.290205 10131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:14.290319 10131 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:28:14.290390 10131 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:28:14.290520 10131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:14.300768 10131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.29.254:42027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:14.302145 10131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:14.303499 10131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:14.316488 10136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:14.316607 10137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:14.316810 10139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:14.317564 10131 server_base.cc:1034] running on GCE node
I20260923 17:28:14.318177 10131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:14.319193 10131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:14.320401 10131 hybrid_clock.cc:648] HybridClock initialized: now 1790184494320345 us; error 53 us; skew 500 ppm
I20260923 17:28:14.320739 10131 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:14 dist-test-slave-p7jt krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184494, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:14.330081 10131 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:28:14.331923 10131 webserver.cc:460] Webserver started at http://127.8.29.193:37265/ using document root <none> and password file <none>
I20260923 17:28:14.332705 10131 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:14.332837 10131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:14.333189 10131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:14.336341 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "7c1c87fb81f94dd88b1acba0e4dd325a"
format_stamp: "Formatted at 2026-09-23 17:28:14 on dist-test-slave-p7jt"
server_key: "228cb6406c6e6b13f3131dcd0e0432c1"
server_key_iv: "14cea5bbf421cb6eae4ad4ffda27b334"
server_key_version: "encryptionkey@0"
I20260923 17:28:14.337225 10131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "7c1c87fb81f94dd88b1acba0e4dd325a"
format_stamp: "Formatted at 2026-09-23 17:28:14 on dist-test-slave-p7jt"
server_key: "228cb6406c6e6b13f3131dcd0e0432c1"
server_key_iv: "14cea5bbf421cb6eae4ad4ffda27b334"
server_key_version: "encryptionkey@0"
I20260923 17:28:14.342972 10131 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:28:14.346872 10146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:14.348485 10131 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:14.348665 10131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "7c1c87fb81f94dd88b1acba0e4dd325a"
format_stamp: "Formatted at 2026-09-23 17:28:14 on dist-test-slave-p7jt"
server_key: "228cb6406c6e6b13f3131dcd0e0432c1"
server_key_iv: "14cea5bbf421cb6eae4ad4ffda27b334"
server_key_version: "encryptionkey@0"
I20260923 17:28:14.348873 10131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:14.363252 10131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:14.367722 10131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:14.368022 10131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:14.370863 10131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:14.370972 10131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:14.371119 10131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:14.371197 10131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:28:14 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184494, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:14.395761 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:14.371463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:34763 (local address 127.8.29.254:42027)
0923 17:28:14.372104 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:14.372111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:14.373334 (+ 1223us) server_negotiation.cc:407] Connection header received
0923 17:28:14.375745 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:14.375752 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:14.375859 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:14.375944 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:14.378596 (+ 2652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.379806 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.382449 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.383003 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.383161 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:14.388228 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:14.388236 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:14.388241 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:14.388286 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:14.391561 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:14.392763 (+ 1202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:14.392769 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:14.392772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:14.392944 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:14.393546 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:14.393551 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:14.393555 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:14.393925 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:14.394094 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:14.395120 (+ 1026us) server_negotiation.cc:299] Negotiation successful
0923 17:28:14.395381 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":379,"threads_started":1}
I20260923 17:28:14.397027 10156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:14.372178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:42027 (local address 127.8.29.193:34763)
0923 17:28:14.373126 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:14.373177 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:14.375355 (+ 2178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:14.376554 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:14.376563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:14.377385 (+ 822us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:14.378314 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:14.378350 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.380039 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.380045 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:14.382268 (+ 2223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:14.382285 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.383301 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.383311 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:14.383689 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:14.384605 (+ 916us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:14.384641 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:14.387945 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:14.391784 (+ 3839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:14.391796 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:14.391822 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:14.392550 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:14.393073 (+ 523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:14.393078 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:14.393082 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:14.393421 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:14.394098 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:14.394108 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:14.394617 (+ 509us) client_negotiation.cc:769] Sending connection context
0923 17:28:14.395016 (+ 399us) client_negotiation.cc:240] Negotiation successful
0923 17:28:14.395641 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":642,"thread_start_us":408,"threads_started":1}
I20260923 17:28:14.415298 10131 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:40757
I20260923 17:28:14.415382 10266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:40757 every 8 connection(s)
I20260923 17:28:14.416764 10131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:14.419032 10267 heartbeater.cc:347] Connected to a master server at 127.8.29.254:42027
I20260923 17:28:14.419269 10267 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:14.419955 10267 heartbeater.cc:511] Master 127.8.29.254:42027 requested a full tablet report, sending...
I20260923 17:28:14.421011 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:14.404239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:60255 (local address 127.8.29.254:42027)
0923 17:28:14.404392 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:14.404397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:14.407156 (+ 2759us) server_negotiation.cc:407] Connection header received
0923 17:28:14.407373 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:14.407377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:14.407447 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:14.407530 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:14.408754 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.409668 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.411702 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.412497 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.412671 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:14.415030 (+ 2359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:14.415036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:14.415040 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:14.415084 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:14.418149 (+ 3065us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:14.418982 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:14.418987 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:14.418990 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:14.419137 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:14.419538 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:14.419543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:14.419545 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:14.419833 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:14.419988 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:14.420433 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 17:28:14.420656 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 17:28:14.421074 10242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:14.403857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:42027 (local address 127.8.29.193:60255)
0923 17:28:14.407035 (+ 3178us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:14.407058 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:14.407231 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:14.407601 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:14.407606 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:14.408104 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:14.408590 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:14.408601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.409849 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.409855 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:14.411492 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:14.411509 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.412949 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.412954 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:14.413140 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:14.413838 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:14.413855 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:14.414813 (+ 958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:14.418308 (+ 3495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:14.418313 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:14.418318 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:14.418845 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:14.419240 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:14.419246 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:14.419249 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:14.419419 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:14.419969 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:14.419974 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:14.420159 (+ 185us) client_negotiation.cc:769] Sending connection context
0923 17:28:14.420277 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:28:14.420626 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3018,"thread_start_us":2860,"threads_started":1}
I20260923 17:28:14.422235 10060 ts_manager.cc:194] Registered new tserver with Master: 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757)
I20260923 17:28:14.422370 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 10131
I20260923 17:28:14.422636 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:14.423493 8311 external_mini_cluster.cc:1428] Setting key 08a69c6a46444139d93937e7242e18eb
I20260923 17:28:14.423828 10060 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:34763
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:28:14.466573 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:42027
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:14.794044 10271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:14.794338 10271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:14.794494 10271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:14.801841 10271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:14.801973 10271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:14.802085 10271 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:28:14.802160 10271 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:28:14.802244 10271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:14.812677 10271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.29.254:42027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:14.814093 10271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:14.815516 10271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:14.828485 10276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:14.828516 10277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:14.828544 10279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:14.829456 10271 server_base.cc:1034] running on GCE node
I20260923 17:28:14.830137 10271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:14.831149 10271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:14.832376 10271 hybrid_clock.cc:648] HybridClock initialized: now 1790184494832315 us; error 47 us; skew 500 ppm
I20260923 17:28:14.832715 10271 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:14 dist-test-slave-p7jt krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184494, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:14.836133 10271 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:28:14.837786 10271 webserver.cc:460] Webserver started at http://127.8.29.194:43295/ using document root <none> and password file <none>
I20260923 17:28:14.838546 10271 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:14.838668 10271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:14.838985 10271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:14.842139 10271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "506d8695dfb84605a87db878f5708404"
format_stamp: "Formatted at 2026-09-23 17:28:14 on dist-test-slave-p7jt"
server_key: "20657784240bcf7f988b778a65a6c002"
server_key_iv: "db319114732aa0508fd626f3644054a0"
server_key_version: "encryptionkey@0"
I20260923 17:28:14.843135 10271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "506d8695dfb84605a87db878f5708404"
format_stamp: "Formatted at 2026-09-23 17:28:14 on dist-test-slave-p7jt"
server_key: "20657784240bcf7f988b778a65a6c002"
server_key_iv: "db319114732aa0508fd626f3644054a0"
server_key_version: "encryptionkey@0"
I20260923 17:28:14.848942 10271 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:14.852905 10286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:14.854519 10271 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:28:14.854689 10271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "506d8695dfb84605a87db878f5708404"
format_stamp: "Formatted at 2026-09-23 17:28:14 on dist-test-slave-p7jt"
server_key: "20657784240bcf7f988b778a65a6c002"
server_key_iv: "db319114732aa0508fd626f3644054a0"
server_key_version: "encryptionkey@0"
I20260923 17:28:14.854902 10271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:14.875418 10271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:14.880300 10271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:14.880656 10271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:14.883745 10271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:14.883872 10271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:14.883996 10271 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:14.884070 10271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:28:14 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184494, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:14.910158 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:14.884708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:51361 (local address 127.8.29.254:42027)
0923 17:28:14.884979 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:14.884988 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:14.886699 (+ 1711us) server_negotiation.cc:407] Connection header received
0923 17:28:14.889750 (+ 3051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:14.889757 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:14.889862 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:14.889973 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:14.892400 (+ 2427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.893432 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.895746 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.896346 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.896505 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:14.902600 (+ 6095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:14.902609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:14.902614 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:14.902666 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:14.906129 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:14.907520 (+ 1391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:14.907527 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:14.907530 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:14.907699 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:14.908273 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:14.908279 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:14.908282 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:14.908602 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:14.908745 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:14.909579 (+ 834us) server_negotiation.cc:299] Negotiation successful
0923 17:28:14.909818 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 17:28:14.911685 10295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:14.885449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:42027 (local address 127.8.29.194:51361)
0923 17:28:14.886436 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:14.886528 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:14.889264 (+ 2736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:14.890315 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:14.890339 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:14.891228 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:14.892159 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:14.892192 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.893614 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.893618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:14.895561 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:14.895581 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.896667 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.896680 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:14.897069 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:14.898051 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:14.898091 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:14.902178 (+ 4087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:14.906359 (+ 4181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:14.906375 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:14.906404 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:14.907286 (+ 882us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:14.907876 (+ 590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:14.907880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:14.907883 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:14.908125 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:14.908747 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:14.908757 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:14.909185 (+ 428us) client_negotiation.cc:769] Sending connection context
0923 17:28:14.909437 (+ 252us) client_negotiation.cc:240] Negotiation successful
0923 17:28:14.910024 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":385,"threads_started":1}
I20260923 17:28:14.933032 10271 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:45835
I20260923 17:28:14.933086 10405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:45835 every 8 connection(s)
I20260923 17:28:14.933748 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:14.918656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:60863 (local address 127.8.29.254:42027)
0923 17:28:14.918843 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:14.918848 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:14.919694 (+ 846us) server_negotiation.cc:407] Connection header received
0923 17:28:14.919870 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:14.919876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:14.919944 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:14.920030 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:14.921055 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.921804 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.923298 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.923875 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.924011 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:14.926697 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:14.926703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:14.926707 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:14.926745 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:14.930347 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:14.931368 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:14.931375 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:14.931379 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:14.931604 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:14.932074 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:14.932078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:14.932080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:14.932411 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:14.932578 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:14.933146 (+ 568us) server_negotiation.cc:299] Negotiation successful
0923 17:28:14.933375 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:28:14.933888 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:14.918679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:42027 (local address 127.8.29.194:60863)
0923 17:28:14.919543 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:14.919562 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:14.919727 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:14.920082 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:14.920087 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:14.920577 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:14.920879 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:14.920889 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.921974 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.921979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:14.923134 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:14.923148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:14.924120 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:14.924128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:14.924384 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:14.925140 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:14.925172 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:14.926333 (+ 1161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:14.930578 (+ 4245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:14.930585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:14.930590 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:14.931178 (+ 588us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:14.931763 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:14.931769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:14.931773 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:14.931934 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:14.932560 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:14.932564 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:14.932829 (+ 265us) client_negotiation.cc:769] Sending connection context
0923 17:28:14.933009 (+ 180us) client_negotiation.cc:240] Negotiation successful
0923 17:28:14.933453 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"thread_start_us":439,"threads_started":1}
I20260923 17:28:14.935009 10271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:14.937335 10406 heartbeater.cc:347] Connected to a master server at 127.8.29.254:42027
I20260923 17:28:14.938290 10406 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:14.938998 10406 heartbeater.cc:511] Master 127.8.29.254:42027 requested a full tablet report, sending...
I20260923 17:28:14.940412 10059 ts_manager.cc:194] Registered new tserver with Master: 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835)
I20260923 17:28:14.941314 10059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:51361
I20260923 17:28:14.945045 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 10271
I20260923 17:28:14.945235 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:14.945706 8311 external_mini_cluster.cc:1428] Setting key 0a4f5dae0e21e555b2a15da04f8cea28
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:28:14.990330 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:42027
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:44437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:15.317342 10410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:15.317634 10410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:15.317764 10410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:15.325224 10410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:15.325353 10410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:15.325472 10410 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 17:28:15.325541 10410 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:28:15.325621 10410 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:15.335747 10410 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:44437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.29.254:42027
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:15.337143 10410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:15.338577 10410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:15.351681 10416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:15.351694 10418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:15.351694 10415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:15.352589 10410 server_base.cc:1034] running on GCE node
I20260923 17:28:15.353169 10410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:15.354138 10410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:15.355355 10410 hybrid_clock.cc:648] HybridClock initialized: now 1790184495355259 us; error 94 us; skew 500 ppm
I20260923 17:28:15.355701 10410 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:15 dist-test-slave-p7jt krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184495, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:15.358850 10410 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:28:15.360414 10410 webserver.cc:460] Webserver started at http://127.8.29.195:35341/ using document root <none> and password file <none>
I20260923 17:28:15.361092 10410 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:15.361196 10410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:15.361513 10410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:15.364527 10410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "515be6f38e964ac4899c9224a0db3430"
format_stamp: "Formatted at 2026-09-23 17:28:15 on dist-test-slave-p7jt"
server_key: "a5a01d13453173453afdb8b6eb49caec"
server_key_iv: "4cabe7c8807c9c38d7b5486b816df987"
server_key_version: "encryptionkey@0"
I20260923 17:28:15.365302 10410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "515be6f38e964ac4899c9224a0db3430"
format_stamp: "Formatted at 2026-09-23 17:28:15 on dist-test-slave-p7jt"
server_key: "a5a01d13453173453afdb8b6eb49caec"
server_key_iv: "4cabe7c8807c9c38d7b5486b816df987"
server_key_version: "encryptionkey@0"
I20260923 17:28:15.371371 10410 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260923 17:28:15.375404 10425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:15.376971 10410 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20260923 17:28:15.377135 10410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "515be6f38e964ac4899c9224a0db3430"
format_stamp: "Formatted at 2026-09-23 17:28:15 on dist-test-slave-p7jt"
server_key: "a5a01d13453173453afdb8b6eb49caec"
server_key_iv: "4cabe7c8807c9c38d7b5486b816df987"
server_key_version: "encryptionkey@0"
I20260923 17:28:15.377357 10410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:15.393498 10410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:15.397496 10410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:15.397773 10410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:15.400697 10410 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:15.400803 10410 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:15.400923 10410 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:15.401000 10410 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:28:15 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184495, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:15.425047 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.401859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:42027 (local address 127.8.29.195:46691)
0923 17:28:15.402797 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:15.402854 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:15.404788 (+ 1934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:15.405692 (+ 904us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:15.405708 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:15.406719 (+ 1011us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:15.407974 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.408018 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.409494 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.409499 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.411862 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.411886 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.413293 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.413303 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.413617 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.414482 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:15.414523 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:15.417032 (+ 2509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:15.420356 (+ 3324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:15.420367 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:15.420385 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:15.420844 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:15.421260 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:15.421265 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:15.421269 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:15.421456 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:15.422050 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:15.422058 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:15.422439 (+ 381us) client_negotiation.cc:769] Sending connection context
0923 17:28:15.422815 (+ 376us) client_negotiation.cc:240] Negotiation successful
0923 17:28:15.423324 (+ 509us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":378,"threads_started":1}
I20260923 17:28:15.425791 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.401282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:46691 (local address 127.8.29.254:42027)
0923 17:28:15.401489 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:15.401496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:15.403002 (+ 1506us) server_negotiation.cc:407] Connection header received
0923 17:28:15.405117 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:15.405122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:15.405222 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:15.405323 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:15.408282 (+ 2959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.409235 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.412269 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.412835 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.412973 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.417326 (+ 4353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:15.417334 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:15.417338 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:15.417384 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:15.420186 (+ 2802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:15.420994 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:15.420998 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:15.421001 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:15.421141 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:15.421586 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:15.421589 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:15.421591 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:15.421912 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:15.422058 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:15.425165 (+ 3107us) server_negotiation.cc:299] Negotiation successful
0923 17:28:15.425441 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:28:15.427270 10267 heartbeater.cc:503] Master 127.8.29.254:42027 was elected leader, sending a full tablet report...
I20260923 17:28:15.448907 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.431858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:42027 (local address 127.8.29.195:51533)
0923 17:28:15.433695 (+ 1837us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:15.433719 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:15.433907 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:15.434493 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:15.434499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:15.435204 (+ 705us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:15.435738 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.435762 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.437127 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.437133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.438917 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.438937 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.439912 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.439921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.440146 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.440833 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:15.440854 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:15.441657 (+ 803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:15.445624 (+ 3967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:15.445630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:15.445633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:15.446376 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:15.446916 (+ 540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:15.446922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:15.446926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:15.447115 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:15.447825 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:15.447831 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:15.448041 (+ 210us) client_negotiation.cc:769] Sending connection context
0923 17:28:15.448187 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 17:28:15.448580 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1650,"thread_start_us":1482,"threads_started":1}
I20260923 17:28:15.449038 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.432013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:51533 (local address 127.8.29.254:42027)
0923 17:28:15.432232 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:15.432240 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:15.433847 (+ 1607us) server_negotiation.cc:407] Connection header received
0923 17:28:15.434027 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:15.434032 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:15.434159 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:15.434260 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:15.435987 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.436971 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.439173 (+ 2202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.439677 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.439827 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.441920 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:15.441927 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:15.441931 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:15.441974 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:15.445402 (+ 3428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:15.446609 (+ 1207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:15.446616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:15.446620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:15.446756 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:15.447293 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:15.447298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:15.447300 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:15.447694 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:15.447861 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:15.448368 (+ 507us) server_negotiation.cc:299] Negotiation successful
0923 17:28:15.448655 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 17:28:15.459828 10410 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:33927
I20260923 17:28:15.459873 10544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:33927 every 8 connection(s)
I20260923 17:28:15.461256 10410 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:15.463757 10545 heartbeater.cc:347] Connected to a master server at 127.8.29.254:42027
I20260923 17:28:15.464030 10545 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:15.464715 10545 heartbeater.cc:511] Master 127.8.29.254:42027 requested a full tablet report, sending...
I20260923 17:28:15.466217 10059 ts_manager.cc:194] Registered new tserver with Master: 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927)
I20260923 17:28:15.467097 10059 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:46691
I20260923 17:28:15.469877 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 10410
I20260923 17:28:15.470072 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:15.470515 8311 external_mini_cluster.cc:1428] Setting key 8f8a37396f1b596f10d7929cc163e0c6
I20260923 17:28:15.473869 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:28:15 dist-test-slave-p7jt krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:28:15 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184495, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:15.521917 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.506266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.8.29.254:42027)
0923 17:28:15.506524 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:15.506530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:15.506840 (+ 310us) server_negotiation.cc:407] Connection header received
0923 17:28:15.507052 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:15.507057 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:15.507134 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:15.507233 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:15.508751 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.509631 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.511281 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.511738 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.511859 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.516093 (+ 4234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:15.516100 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:15.516103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:15.516146 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:15.518807 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:15.519735 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:15.519740 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:15.519742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:15.519890 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:15.520395 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:15.520399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:15.520401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:15.520770 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:15.520976 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:15.521365 (+ 389us) server_negotiation.cc:299] Negotiation successful
0923 17:28:15.521548 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 17:28:15.526317 10059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:15.533417 10059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:15.556912 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.546079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:45835 (local address 127.0.0.1:56486)
0923 17:28:15.547071 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:15.547122 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:15.547298 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:15.548348 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:15.548356 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:15.548389 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:15.548833 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.548845 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.550852 (+ 2007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.550859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.553452 (+ 2593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.553469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.555792 (+ 2323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.555799 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.556060 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.556116 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 17:28:15.556258 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 17:28:15.556475 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":1099,"threads_started":1}
I20260923 17:28:15.557672 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.546266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56486 (local address 127.8.29.194:45835)
0923 17:28:15.546923 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:15.546932 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:15.547519 (+ 587us) server_negotiation.cc:407] Connection header received
0923 17:28:15.547623 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:15.547631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:15.547911 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:15.548117 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:15.549061 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.550680 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.553664 (+ 2984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.555249 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.555402 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.555644 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:15.557214 (+ 1570us) server_negotiation.cc:299] Negotiation successful
0923 17:28:15.557396 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":717,"threads_started":1}
I20260923 17:28:15.563411 10336 tablet_service.cc:1467] Processing CreateTablet for tablet 123e8db2745a41cfb81159673a067815 (DEFAULT_TABLE table=test-table [id=7b41c497c4024e94b9da190bbad622ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:15.564584 10558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.546731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:33927 (local address 127.0.0.1:45388)
0923 17:28:15.557165 (+ 10434us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:15.557183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:15.557284 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:15.558005 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:15.558010 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:15.558035 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:15.558220 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.558231 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.560100 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.560104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.561854 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.561870 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.563701 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.563707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.563910 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.563946 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:28:15.564049 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 17:28:15.564299 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10347,"thread_start_us":10173,"threads_started":1}
I20260923 17:28:15.564603 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.547257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45388 (local address 127.8.29.195:33927)
0923 17:28:15.556825 (+ 9568us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:15.556836 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:15.557401 (+ 565us) server_negotiation.cc:407] Connection header received
0923 17:28:15.557510 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:15.557519 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:15.557799 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:15.558031 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:15.558364 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.559907 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.562083 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.563530 (+ 1447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.563663 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.563902 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:15.564110 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 17:28:15.564283 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9083,"thread_start_us":435,"threads_started":1}
I20260923 17:28:15.565028 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123e8db2745a41cfb81159673a067815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:15.568832 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.548215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.8.29.193:40757)
0923 17:28:15.559041 (+ 10826us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:15.559050 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:15.559080 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:28:15.559171 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:15.559178 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:15.559556 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:15.559758 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:15.562587 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.564458 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.566336 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.567776 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.567935 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.568207 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:15.568366 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 17:28:15.568540 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10570,"thread_start_us":10433,"threads_started":1}
I20260923 17:28:15.568673 10475 tablet_service.cc:1467] Processing CreateTablet for tablet 123e8db2745a41cfb81159673a067815 (DEFAULT_TABLE table=test-table [id=7b41c497c4024e94b9da190bbad622ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:15.569967 10560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.547935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.0.0.1:42288)
0923 17:28:15.557671 (+ 9736us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:15.557691 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:15.558301 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:15.559764 (+ 1463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:15.559769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:15.559795 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:15.560171 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.560185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.564658 (+ 4473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.564663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.566125 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.566141 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.567992 (+ 1851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.567999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.568169 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.568204 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:28:15.568300 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:28:15.569652 (+ 1352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9622,"thread_start_us":11141,"threads_started":1}
I20260923 17:28:15.570235 10475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123e8db2745a41cfb81159673a067815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:15.574060 10197 tablet_service.cc:1467] Processing CreateTablet for tablet 123e8db2745a41cfb81159673a067815 (DEFAULT_TABLE table=test-table [id=7b41c497c4024e94b9da190bbad622ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:15.575723 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 123e8db2745a41cfb81159673a067815. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:15.580499 10564 tablet_bootstrap.cc:492] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: Bootstrap starting.
I20260923 17:28:15.582640 10565 tablet_bootstrap.cc:492] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: Bootstrap starting.
I20260923 17:28:15.584196 10564 tablet_bootstrap.cc:654] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:15.586030 10564 log.cc:824] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:15.586242 10565 tablet_bootstrap.cc:654] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:15.587497 10565 log.cc:824] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:15.588929 10564 tablet_bootstrap.cc:492] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: No bootstrap required, opened a new log
I20260923 17:28:15.589186 10564 ts_tablet_manager.cc:1397] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20260923 17:28:15.590579 10565 tablet_bootstrap.cc:492] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: No bootstrap required, opened a new log
I20260923 17:28:15.590889 10565 ts_tablet_manager.cc:1397] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 17:28:15.591439 10568 tablet_bootstrap.cc:492] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: Bootstrap starting.
I20260923 17:28:15.595185 10568 tablet_bootstrap.cc:654] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:15.596520 10568 log.cc:824] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:15.598937 10565 raft_consensus.cc:348] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.599419 10565 raft_consensus.cc:374] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:15.599524 10565 raft_consensus.cc:729] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515be6f38e964ac4899c9224a0db3430, State: Initialized, Role: FOLLOWER
I20260923 17:28:15.599354 10564 raft_consensus.cc:348] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.599808 10564 raft_consensus.cc:374] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:15.599897 10564 raft_consensus.cc:729] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506d8695dfb84605a87db878f5708404, State: Initialized, Role: FOLLOWER
I20260923 17:28:15.600275 10565 consensus_queue.cc:260] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.600440 10564 consensus_queue.cc:260] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.601437 10568 tablet_bootstrap.cc:492] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: No bootstrap required, opened a new log
I20260923 17:28:15.601464 10545 heartbeater.cc:503] Master 127.8.29.254:42027 was elected leader, sending a full tablet report...
I20260923 17:28:15.601732 10568 ts_tablet_manager.cc:1397] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260923 17:28:15.602205 10565 ts_tablet_manager.cc:1428] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260923 17:28:15.603209 10406 heartbeater.cc:503] Master 127.8.29.254:42027 was elected leader, sending a full tablet report...
I20260923 17:28:15.607337 10564 ts_tablet_manager.cc:1428] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: Time spent starting tablet: real 0.018s user 0.010s sys 0.006s
I20260923 17:28:15.610931 10568 raft_consensus.cc:348] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.611335 10568 raft_consensus.cc:374] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:15.611428 10568 raft_consensus.cc:729] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1c87fb81f94dd88b1acba0e4dd325a, State: Initialized, Role: FOLLOWER
I20260923 17:28:15.611984 10568 consensus_queue.cc:260] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.613662 10568 ts_tablet_manager.cc:1428] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260923 17:28:15.620424 10570 raft_consensus.cc:482] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:15.620733 10570 raft_consensus.cc:504] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.622354 10570 leader_election.cc:288] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835), 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757)
I20260923 17:28:15.628398 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.622828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:57877 (local address 127.8.29.194:45835)
0923 17:28:15.623007 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:15.623012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:15.623075 (+ 63us) server_negotiation.cc:407] Connection header received
0923 17:28:15.623269 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:15.623274 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:15.623349 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:15.623444 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:15.623917 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.624933 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.626767 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.627538 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.627680 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.627754 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:15.628028 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 17:28:15.628152 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:28:15.628394 10434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.622779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:45835 (local address 127.8.29.195:57877)
0923 17:28:15.622980 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:15.623009 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:15.623133 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:15.623505 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:15.623510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:15.623535 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:15.623759 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.623774 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.625057 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.625063 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.626618 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.626631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.627676 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.627682 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.627869 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.627895 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:28:15.627974 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 17:28:15.628109 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 17:28:15.629379 10573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.623150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.8.29.195:39833)
0923 17:28:15.624047 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:15.624064 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:15.624179 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:15.624501 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:15.624504 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:15.624529 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:15.624777 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.624785 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.625962 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.625967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.627406 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.627418 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.628760 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.628767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.628929 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.628961 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:28:15.629056 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:28:15.629154 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":767,"thread_start_us":569,"threads_started":1}
I20260923 17:28:15.629467 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.623361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:39833 (local address 127.8.29.193:40757)
0923 17:28:15.623580 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:15.623587 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:15.624159 (+ 572us) server_negotiation.cc:407] Connection header received
0923 17:28:15.624300 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:15.624305 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:15.624390 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:15.624479 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:15.624924 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.625819 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.627554 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.628443 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.628596 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.628660 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:15.629115 (+ 455us) server_negotiation.cc:299] Negotiation successful
0923 17:28:15.629239 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:28:15.629575 10356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123e8db2745a41cfb81159673a067815" candidate_uuid: "515be6f38e964ac4899c9224a0db3430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d8695dfb84605a87db878f5708404" is_pre_election: true
I20260923 17:28:15.630110 10356 raft_consensus.cc:2454] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 515be6f38e964ac4899c9224a0db3430 in term 0.
I20260923 17:28:15.630326 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123e8db2745a41cfb81159673a067815" candidate_uuid: "515be6f38e964ac4899c9224a0db3430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" is_pre_election: true
I20260923 17:28:15.630748 10429 leader_election.cc:302] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506d8695dfb84605a87db878f5708404, 515be6f38e964ac4899c9224a0db3430; no voters:
I20260923 17:28:15.630854 10217 raft_consensus.cc:2454] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 515be6f38e964ac4899c9224a0db3430 in term 0.
I20260923 17:28:15.631110 10570 raft_consensus.cc:2781] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:15.631235 10570 raft_consensus.cc:482] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:15.631346 10570 raft_consensus.cc:3037] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:15.633980 10570 raft_consensus.cc:504] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.634760 10570 leader_election.cc:288] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [CANDIDATE]: Term 1 election: Requested vote from peers 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835), 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757)
I20260923 17:28:15.635411 10356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123e8db2745a41cfb81159673a067815" candidate_uuid: "515be6f38e964ac4899c9224a0db3430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d8695dfb84605a87db878f5708404"
I20260923 17:28:15.635635 10356 raft_consensus.cc:3037] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:15.635591 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "123e8db2745a41cfb81159673a067815" candidate_uuid: "515be6f38e964ac4899c9224a0db3430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a"
I20260923 17:28:15.635812 10217 raft_consensus.cc:3037] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:15.637909 10356 raft_consensus.cc:2454] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 515be6f38e964ac4899c9224a0db3430 in term 1.
I20260923 17:28:15.637996 10217 raft_consensus.cc:2454] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 515be6f38e964ac4899c9224a0db3430 in term 1.
I20260923 17:28:15.638339 10429 leader_election.cc:302] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506d8695dfb84605a87db878f5708404, 515be6f38e964ac4899c9224a0db3430; no voters:
I20260923 17:28:15.638674 10570 raft_consensus.cc:2781] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:15.639117 10570 raft_consensus.cc:686] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 1 LEADER]: Becoming Leader. State: Replica: 515be6f38e964ac4899c9224a0db3430, State: Running, Role: LEADER
I20260923 17:28:15.639590 10570 consensus_queue.cc:237] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.644793 10059 catalog_manager.cc:5495] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 reported cstate change: term changed from 0 to 1, leader changed from <none> to 515be6f38e964ac4899c9224a0db3430 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "515be6f38e964ac4899c9224a0db3430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } health_report { overall_health: HEALTHY } } }
W20260923 17:28:15.669732 10268 tablet.cc:2369] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:28:15.687322 10407 tablet.cc:2369] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 17:28:15 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184492, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:15.705958 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.693010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:42027 (local address 127.0.0.1:60750)
0923 17:28:15.693592 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:15.693607 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:15.693768 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:15.694080 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:15.694084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:15.694829 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:15.695086 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.695096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.696086 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.696090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.697058 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:15.697070 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.697791 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.697797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:15.697945 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.698589 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:15.698606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:15.700842 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:15.703695 (+ 2853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:15.703706 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:15.703712 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:15.704133 (+ 421us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:15.704455 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:15.704458 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:15.704460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:15.704594 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:15.705060 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:15.705067 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:15.705221 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 17:28:15.705367 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 17:28:15.705593 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":328,"threads_started":1}
I20260923 17:28:15.705989 10155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:15.692997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60750 (local address 127.8.29.254:42027)
0923 17:28:15.693211 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:15.693217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:15.693716 (+ 499us) server_negotiation.cc:407] Connection header received
0923 17:28:15.693879 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:15.693884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:15.693951 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:15.694030 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:15.695234 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.695947 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.697197 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:15.697605 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:15.697714 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:15.701002 (+ 3288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:15.701007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:15.701011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:15.701041 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:15.703569 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:15.704235 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:15.704239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:15.704241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:15.704364 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:15.704682 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:15.704685 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:15.704687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:15.704956 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:15.705060 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:15.705448 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 17:28:15.705706 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":87,"server-negotiator.queue_time_us":61}
I20260923 17:28:15.711040 10059 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:60750:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:28:15.712065 10059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260923 17:28:15.712996 10546 tablet.cc:2369] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:15.719048 10197 tablet_service.cc:1467] Processing CreateTablet for tablet 4fced77ff1d74daba9d6340ab5ad7ba3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=139d3bb752eb409eb67a5c6499c98384]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:28:15.719065 10336 tablet_service.cc:1467] Processing CreateTablet for tablet 4fced77ff1d74daba9d6340ab5ad7ba3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=139d3bb752eb409eb67a5c6499c98384]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:28:15.719065 10475 tablet_service.cc:1467] Processing CreateTablet for tablet 4fced77ff1d74daba9d6340ab5ad7ba3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=139d3bb752eb409eb67a5c6499c98384]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:28:15.719974 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fced77ff1d74daba9d6340ab5ad7ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:15.719985 10475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fced77ff1d74daba9d6340ab5ad7ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:15.719985 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4fced77ff1d74daba9d6340ab5ad7ba3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:15.725903 10568 tablet_bootstrap.cc:492] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: Bootstrap starting.
I20260923 17:28:15.727320 10568 tablet_bootstrap.cc:654] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:15.727648 10565 tablet_bootstrap.cc:492] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: Bootstrap starting.
I20260923 17:28:15.728627 10564 tablet_bootstrap.cc:492] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: Bootstrap starting.
I20260923 17:28:15.729377 10565 tablet_bootstrap.cc:654] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:15.730511 10564 tablet_bootstrap.cc:654] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:15.732789 10564 tablet_bootstrap.cc:492] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: No bootstrap required, opened a new log
I20260923 17:28:15.732939 10564 ts_tablet_manager.cc:1397] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20260923 17:28:15.732997 10568 tablet_bootstrap.cc:492] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: No bootstrap required, opened a new log
I20260923 17:28:15.733120 10565 tablet_bootstrap.cc:492] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: No bootstrap required, opened a new log
I20260923 17:28:15.733172 10568 ts_tablet_manager.cc:1397] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260923 17:28:15.733266 10565 ts_tablet_manager.cc:1397] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260923 17:28:15.733665 10564 raft_consensus.cc:348] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.733882 10564 raft_consensus.cc:374] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:15.733968 10564 raft_consensus.cc:729] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506d8695dfb84605a87db878f5708404, State: Initialized, Role: FOLLOWER
I20260923 17:28:15.733940 10568 raft_consensus.cc:348] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.733999 10565 raft_consensus.cc:348] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.734213 10565 raft_consensus.cc:374] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:15.734210 10568 raft_consensus.cc:374] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:15.734303 10565 raft_consensus.cc:729] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515be6f38e964ac4899c9224a0db3430, State: Initialized, Role: FOLLOWER
I20260923 17:28:15.734302 10568 raft_consensus.cc:729] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1c87fb81f94dd88b1acba0e4dd325a, State: Initialized, Role: FOLLOWER
I20260923 17:28:15.734215 10564 consensus_queue.cc:260] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.734545 10568 consensus_queue.cc:260] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.734887 10564 ts_tablet_manager.cc:1428] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:15.734555 10565 consensus_queue.cc:260] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:15.734702 10571 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:15.735071 10571 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:15.735073 10568 ts_tablet_manager.cc:1428] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 17:28:15.735332 10565 ts_tablet_manager.cc:1428] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:15.735493 10572 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:15.735659 10570 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:15.735865 10572 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:15.735948 10570 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:16.036238 10572 raft_consensus.cc:482] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:16.036470 10572 raft_consensus.cc:504] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.037969 10572 leader_election.cc:288] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835), 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927)
I20260923 17:28:16.039829 10571 raft_consensus.cc:482] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:16.040104 10571 raft_consensus.cc:504] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.041965 10571 leader_election.cc:288] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757), 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927)
I20260923 17:28:16.047196 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.038491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:58103 (local address 127.8.29.195:33927)
0923 17:28:16.038697 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.038704 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.039307 (+ 603us) server_negotiation.cc:407] Connection header received
0923 17:28:16.039493 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.039499 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.039576 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.039659 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:16.040540 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.041490 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.044143 (+ 2653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.045550 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.045787 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.045884 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.046657 (+ 773us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.046878 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:28:16.048226 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.038382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:33927 (local address 127.8.29.193:58103)
0923 17:28:16.039164 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:16.039183 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:16.039348 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:16.039746 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:16.039750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:16.039774 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:16.040295 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.040308 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.041594 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.041598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.043924 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.043945 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.046117 (+ 2172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.046124 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.046414 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.046470 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 17:28:16.047631 (+ 1161us) client_negotiation.cc:240] Negotiation successful
0923 17:28:16.047808 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"mutex_wait_us":21,"thread_start_us":421,"threads_started":1}
I20260923 17:28:16.049228 10500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fced77ff1d74daba9d6340ab5ad7ba3" candidate_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515be6f38e964ac4899c9224a0db3430" is_pre_election: true
I20260923 17:28:16.049755 10500 raft_consensus.cc:2454] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c1c87fb81f94dd88b1acba0e4dd325a in term 0.
I20260923 17:28:16.050504 10147 leader_election.cc:302] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515be6f38e964ac4899c9224a0db3430, 7c1c87fb81f94dd88b1acba0e4dd325a; no voters:
I20260923 17:28:16.052983 10572 raft_consensus.cc:2781] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:16.053135 10572 raft_consensus.cc:482] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:16.053229 10572 raft_consensus.cc:3037] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:16.056620 10572 raft_consensus.cc:504] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.057911 10500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fced77ff1d74daba9d6340ab5ad7ba3" candidate_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515be6f38e964ac4899c9224a0db3430"
I20260923 17:28:16.058043 10559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.038398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:34989 (local address 127.8.29.194:45835)
0923 17:28:16.038598 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.038604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.039513 (+ 909us) server_negotiation.cc:407] Connection header received
0923 17:28:16.039704 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.039709 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.039830 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.039921 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:16.040657 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.041431 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.051102 (+ 9671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.052528 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.052754 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.052838 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.057557 (+ 4719us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.057728 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:28:16.058140 10500 raft_consensus.cc:3037] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:16.058384 10572 leader_election.cc:288] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [CANDIDATE]: Term 1 election: Requested vote from peers 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835), 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927)
I20260923 17:28:16.059144 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.038390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:45835 (local address 127.8.29.193:34989)
0923 17:28:16.039395 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:16.039411 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:16.039571 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:16.040072 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:16.040076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:16.040096 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:16.040465 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.040476 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.041581 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.041586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.050337 (+ 8751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.050353 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.057121 (+ 6768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.057129 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.057317 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.057355 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:28:16.058212 (+ 857us) client_negotiation.cc:240] Negotiation successful
0923 17:28:16.058365 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":874,"mutex_wait_us":225,"thread_start_us":481,"threads_started":1}
I20260923 17:28:16.060194 10356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fced77ff1d74daba9d6340ab5ad7ba3" candidate_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d8695dfb84605a87db878f5708404" is_pre_election: true
I20260923 17:28:16.060474 10356 raft_consensus.cc:2454] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c1c87fb81f94dd88b1acba0e4dd325a in term 0.
I20260923 17:28:16.060897 10355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fced77ff1d74daba9d6340ab5ad7ba3" candidate_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d8695dfb84605a87db878f5708404"
I20260923 17:28:16.061152 10355 raft_consensus.cc:3037] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:16.061224 10500 raft_consensus.cc:2454] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c1c87fb81f94dd88b1acba0e4dd325a in term 1.
I20260923 17:28:16.061655 10147 leader_election.cc:302] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515be6f38e964ac4899c9224a0db3430, 7c1c87fb81f94dd88b1acba0e4dd325a; no voters:
I20260923 17:28:16.061959 10572 raft_consensus.cc:2781] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:16.065306 10355 raft_consensus.cc:2454] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c1c87fb81f94dd88b1acba0e4dd325a in term 1.
I20260923 17:28:16.065912 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.042342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:51609 (local address 127.8.29.193:40757)
0923 17:28:16.042530 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.042536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.058071 (+ 15535us) server_negotiation.cc:407] Connection header received
0923 17:28:16.058525 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.058530 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.058620 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.058698 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:16.059298 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.060334 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.063057 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.064355 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.064508 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.064582 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.065477 (+ 895us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.065622 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:28:16.066627 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.042296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.8.29.194:51609)
0923 17:28:16.057958 (+ 15662us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:16.057979 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:16.058362 (+ 383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:16.058845 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:16.058849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:16.058882 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:16.059134 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.059141 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.060560 (+ 1419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.060564 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.062883 (+ 2319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.062897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.064771 (+ 1874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.064777 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.065006 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.065048 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 17:28:16.066081 (+ 1033us) client_negotiation.cc:240] Negotiation successful
0923 17:28:16.066236 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15487,"mutex_wait_us":1,"thread_start_us":15791,"threads_started":1}
I20260923 17:28:16.067044 10588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.042343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:33927 (local address 127.8.29.194:51535)
0923 17:28:16.058587 (+ 16244us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:16.058605 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:16.058713 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:16.059931 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:16.059935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:16.059959 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:16.060348 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.060361 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.061754 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.061759 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.064152 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.064170 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.066137 (+ 1967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.066144 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.066361 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.066398 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:28:16.066535 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:28:16.066695 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15626,"thread_start_us":15843,"threads_started":1}
I20260923 17:28:16.067158 10572 raft_consensus.cc:686] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 LEADER]: Becoming Leader. State: Replica: 7c1c87fb81f94dd88b1acba0e4dd325a, State: Running, Role: LEADER
I20260923 17:28:16.067337 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fced77ff1d74daba9d6340ab5ad7ba3" candidate_uuid: "506d8695dfb84605a87db878f5708404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" is_pre_election: true
I20260923 17:28:16.067651 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.042714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:51535 (local address 127.8.29.195:33927)
0923 17:28:16.047570 (+ 4856us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.047576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.059518 (+ 11942us) server_negotiation.cc:407] Connection header received
0923 17:28:16.059610 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.059615 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.059689 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.059775 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:16.060541 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.061583 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.064365 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.065684 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.065819 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.065904 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.066582 (+ 678us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.067300 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4736,"thread_start_us":13994,"threads_started":1}
I20260923 17:28:16.067826 10572 consensus_queue.cc:237] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.068343 10500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4fced77ff1d74daba9d6340ab5ad7ba3" candidate_uuid: "506d8695dfb84605a87db878f5708404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515be6f38e964ac4899c9224a0db3430" is_pre_election: true
I20260923 17:28:16.068641 10500 raft_consensus.cc:2379] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 506d8695dfb84605a87db878f5708404 in current term 1: Already voted for candidate 7c1c87fb81f94dd88b1acba0e4dd325a in this term.
I20260923 17:28:16.069303 10287 leader_election.cc:302] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 506d8695dfb84605a87db878f5708404; no voters: 515be6f38e964ac4899c9224a0db3430, 7c1c87fb81f94dd88b1acba0e4dd325a
I20260923 17:28:16.069679 10571 raft_consensus.cc:2726] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 17:28:16.072151 10590 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: TxnStatusTablet state changed. Reason: New leader 7c1c87fb81f94dd88b1acba0e4dd325a. Latest consensus state: current_term: 1 leader_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:16.072494 10590 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:28:16.073225 10592 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:28:16.073508 10592 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:28:16.074054 10057 catalog_manager.cc:5495] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193). New cstate: current_term: 1 leader_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } health_report { overall_health: UNKNOWN } } }
I20260923 17:28:16.104321 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.098121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.0.0.1:42298)
0923 17:28:16.098347 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:16.098365 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:16.098523 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:16.099012 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:16.099015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:16.099025 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:28:16.099182 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.099189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.100610 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.100615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.101573 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.101585 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.102573 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.102577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.102739 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.102803 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.103879 (+ 1076us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.103906 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:28:16.103968 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 17:28:16.104086 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 17:28:16.104422 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.098234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42298 (local address 127.8.29.193:40757)
0923 17:28:16.098438 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.098444 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.098508 (+ 64us) server_negotiation.cc:407] Connection header received
0923 17:28:16.098724 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.098730 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.098836 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.098999 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:16.099313 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.100463 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.101732 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.102394 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.102566 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.102983 (+ 417us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.103753 (+ 770us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.103912 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.103995 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.104131 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:28:16.111956 10500 raft_consensus.cc:1261] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Refusing update from remote peer 7c1c87fb81f94dd88b1acba0e4dd325a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:16.111936 10355 raft_consensus.cc:1261] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Refusing update from remote peer 7c1c87fb81f94dd88b1acba0e4dd325a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:16.112799 10572 consensus_queue.cc:1035] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:16.113229 10590 consensus_queue.cc:1035] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a [LEADER]: Connected to new peer: Peer: permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:16.123689 10571 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: TxnStatusTablet state changed. Reason: New leader 7c1c87fb81f94dd88b1acba0e4dd325a. Latest consensus state: current_term: 1 leader_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:16.124043 10571 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:16.125809 10570 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: TxnStatusTablet state changed. Reason: New leader 7c1c87fb81f94dd88b1acba0e4dd325a. Latest consensus state: current_term: 1 leader_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:16.126113 10570 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:16.127820 10595 mvcc.cc:205] Tried to move back new op lower bound from 7332595696064450560 to 7332595695901147136. Current Snapshot: MvccSnapshot[applied={T|T < 7332595696064450560}]
I20260923 17:28:16.129145 10572 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:16.129444 10572 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:28:16.129707 10572 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:16.129899 10572 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 7c1c87fb81f94dd88b1acba0e4dd325a: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:28:16.130527 10570 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:16.130764 10570 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 515be6f38e964ac4899c9224a0db3430: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:16.137147 10571 tablet_replica.cc:439] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } } }
I20260923 17:28:16.137447 10571 tablet_replica.cc:442] T 4fced77ff1d74daba9d6340ab5ad7ba3 P 506d8695dfb84605a87db878f5708404: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:28:16.146287 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.140383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.8.29.195:33927)
0923 17:28:16.140557 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.140563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.140964 (+ 401us) server_negotiation.cc:407] Connection header received
0923 17:28:16.141212 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.141216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.141310 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.141439 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:16.141933 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.142686 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.144072 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.144505 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.144617 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.145049 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.145675 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.145808 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.145929 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.146018 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 17:28:16.163223 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.157442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:34447 (local address 127.8.29.193:40757)
0923 17:28:16.157679 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.157691 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.158290 (+ 599us) server_negotiation.cc:407] Connection header received
0923 17:28:16.158511 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.158517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.158595 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.158678 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:16.159165 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.159990 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.161357 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.161834 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.161952 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.162411 (+ 459us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.162562 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.162642 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.162787 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.162879 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:28:16.163290 10607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.157393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.8.29.195:34447)
0923 17:28:16.158178 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:16.158195 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:16.158345 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:16.158724 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:16.158728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:16.158741 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:28:16.158971 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.158981 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.160133 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.160139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.161211 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.161224 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.162008 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.162015 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.162153 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.162214 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.162669 (+ 455us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.162680 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:28:16.162746 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 17:28:16.162904 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":457,"threads_started":1}
I20260923 17:28:16.174686 10355 raft_consensus.cc:1261] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Refusing update from remote peer 515be6f38e964ac4899c9224a0db3430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:16.174782 10217 raft_consensus.cc:1261] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Refusing update from remote peer 515be6f38e964ac4899c9224a0db3430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:16.175455 10609 consensus_queue.cc:1035] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:16.175912 10570 consensus_queue.cc:1035] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:16.183346 10597 mvcc.cc:205] Tried to move back new op lower bound from 7332595696323862528 to 7332595694143545344. Current Snapshot: MvccSnapshot[applied={T|T < 7332595696323862528}]
I20260923 17:28:16.219504 10561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.212704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:44745 (local address 127.8.29.195:33927)
0923 17:28:16.212904 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.212909 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.213542 (+ 633us) server_negotiation.cc:407] Connection header received
0923 17:28:16.213738 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.213742 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.213808 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.213884 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:16.214604 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.215786 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.217276 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.217889 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.218033 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.218676 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.218827 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.218920 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.219113 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.219211 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:28:16.219676 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.212683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:33927 (local address 127.8.29.193:44745)
0923 17:28:16.213429 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:16.213445 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:16.213596 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:16.213919 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:16.213924 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:16.213935 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:28:16.214388 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.214401 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.215941 (+ 1540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.215945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.217111 (+ 1166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.217130 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.218104 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.218111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.218338 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.218418 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.218961 (+ 543us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.218979 (+ 18us) client_negotiation.cc:769] Sending connection context
0923 17:28:16.219069 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 17:28:16.219239 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":381,"threads_started":1}
I20260923 17:28:16.260025 10058 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:16.260303 10058 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:16.263967 10058 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a: Sending DeleteTablet for 3 replicas of tablet 123e8db2745a41cfb81159673a067815
I20260923 17:28:16.265240 10336 tablet_service.cc:1514] Processing DeleteTablet for tablet 123e8db2745a41cfb81159673a067815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:16 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:56486
I20260923 17:28:16.265364 10475 tablet_service.cc:1514] Processing DeleteTablet for tablet 123e8db2745a41cfb81159673a067815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:16 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:45388
I20260923 17:28:16.266412 10197 tablet_service.cc:1514] Processing DeleteTablet for tablet 123e8db2745a41cfb81159673a067815 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:16 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:42288
I20260923 17:28:16.267457 10058 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60738:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:16.268730 10058 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:16.279278 10335 tablet_service.cc:1467] Processing CreateTablet for tablet 1621e4dd8d814b3292e91eba6d631222 (DEFAULT_TABLE table=test-table [id=dd6bab960f6f4fe98c197ba8992b8700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:16.279814 10335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1621e4dd8d814b3292e91eba6d631222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:16.283339 10196 tablet_service.cc:1467] Processing CreateTablet for tablet 1621e4dd8d814b3292e91eba6d631222 (DEFAULT_TABLE table=test-table [id=dd6bab960f6f4fe98c197ba8992b8700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:16.283926 10196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1621e4dd8d814b3292e91eba6d631222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:16.287108 10474 tablet_service.cc:1467] Processing CreateTablet for tablet 1621e4dd8d814b3292e91eba6d631222 (DEFAULT_TABLE table=test-table [id=dd6bab960f6f4fe98c197ba8992b8700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:16.287758 10474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1621e4dd8d814b3292e91eba6d631222. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:16.308661 10620 tablet_replica.cc:330] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: stopping tablet replica
I20260923 17:28:16.311343 10623 tablet_bootstrap.cc:492] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: Bootstrap starting.
I20260923 17:28:16.309226 10620 raft_consensus.cc:2229] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:16.313539 10620 raft_consensus.cc:2258] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:16.314173 10623 tablet_bootstrap.cc:654] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:16.316195 10622 tablet_bootstrap.cc:492] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: Bootstrap starting.
I20260923 17:28:16.318502 10622 tablet_bootstrap.cc:654] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:16.319105 10621 tablet_bootstrap.cc:492] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: Bootstrap starting.
I20260923 17:28:16.320971 10621 tablet_bootstrap.cc:654] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:16.322158 10619 tablet_replica.cc:330] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: stopping tablet replica
I20260923 17:28:16.322604 10619 raft_consensus.cc:2229] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:16.318929 10620 ts_tablet_manager.cc:1905] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:16.322980 10619 raft_consensus.cc:2258] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:16.325019 10619 ts_tablet_manager.cc:1905] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:16.320225 10618 tablet_replica.cc:330] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: stopping tablet replica
I20260923 17:28:16.329319 10618 raft_consensus.cc:2229] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:16.330204 10618 raft_consensus.cc:2258] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:16.332736 10619 ts_tablet_manager.cc:1918] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:28:16.332916 10619 log.cc:1196] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/123e8db2745a41cfb81159673a067815
I20260923 17:28:16.333460 10619 ts_tablet_manager.cc:1939] T 123e8db2745a41cfb81159673a067815 P 506d8695dfb84605a87db878f5708404: Deleting consensus metadata
I20260923 17:28:16.334203 10618 ts_tablet_manager.cc:1905] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:16.335450 10047 catalog_manager.cc:4841] TS 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835): tablet 123e8db2745a41cfb81159673a067815 (table test-table [id=7b41c497c4024e94b9da190bbad622ab]) successfully deleted
I20260923 17:28:16.340857 10620 ts_tablet_manager.cc:1918] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:28:16.341135 10623 tablet_bootstrap.cc:492] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: No bootstrap required, opened a new log
I20260923 17:28:16.341181 10620 log.cc:1196] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/123e8db2745a41cfb81159673a067815
I20260923 17:28:16.341315 10623 ts_tablet_manager.cc:1397] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20260923 17:28:16.341909 10620 ts_tablet_manager.cc:1939] T 123e8db2745a41cfb81159673a067815 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting consensus metadata
I20260923 17:28:16.342209 10623 raft_consensus.cc:348] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.342504 10623 raft_consensus.cc:374] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:16.342612 10623 raft_consensus.cc:729] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515be6f38e964ac4899c9224a0db3430, State: Initialized, Role: FOLLOWER
I20260923 17:28:16.343233 10622 tablet_bootstrap.cc:492] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: No bootstrap required, opened a new log
I20260923 17:28:16.343389 10622 ts_tablet_manager.cc:1397] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20260923 17:28:16.343954 10621 tablet_bootstrap.cc:492] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: No bootstrap required, opened a new log
I20260923 17:28:16.344094 10621 ts_tablet_manager.cc:1397] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20260923 17:28:16.344151 10622 raft_consensus.cc:348] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.344378 10622 raft_consensus.cc:374] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:16.344455 10622 raft_consensus.cc:729] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1c87fb81f94dd88b1acba0e4dd325a, State: Initialized, Role: FOLLOWER
I20260923 17:28:16.344864 10621 raft_consensus.cc:348] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.345103 10621 raft_consensus.cc:374] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:16.345204 10621 raft_consensus.cc:729] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506d8695dfb84605a87db878f5708404, State: Initialized, Role: FOLLOWER
I20260923 17:28:16.345757 10621 consensus_queue.cc:260] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.345556 10623 consensus_queue.cc:260] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.347100 10047 catalog_manager.cc:4841] TS 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757): tablet 123e8db2745a41cfb81159673a067815 (table test-table [id=7b41c497c4024e94b9da190bbad622ab]) successfully deleted
I20260923 17:28:16.347658 10623 ts_tablet_manager.cc:1428] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260923 17:28:16.349501 10621 ts_tablet_manager.cc:1428] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260923 17:28:16.350639 10622 consensus_queue.cc:260] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.351290 10622 ts_tablet_manager.cc:1428] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260923 17:28:16.350942 10618 ts_tablet_manager.cc:1918] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:28:16.358783 10618 log.cc:1196] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/123e8db2745a41cfb81159673a067815
I20260923 17:28:16.359310 10618 ts_tablet_manager.cc:1939] T 123e8db2745a41cfb81159673a067815 P 515be6f38e964ac4899c9224a0db3430: Deleting consensus metadata
I20260923 17:28:16.360966 10044 catalog_manager.cc:4841] TS 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927): tablet 123e8db2745a41cfb81159673a067815 (table test-table [id=7b41c497c4024e94b9da190bbad622ab]) successfully deleted
I20260923 17:28:16.542762 10616 raft_consensus.cc:482] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:16.543044 10616 raft_consensus.cc:504] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.543903 10616 leader_election.cc:288] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835), 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927)
I20260923 17:28:16.544570 10356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1621e4dd8d814b3292e91eba6d631222" candidate_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d8695dfb84605a87db878f5708404" is_pre_election: true
I20260923 17:28:16.544682 10500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1621e4dd8d814b3292e91eba6d631222" candidate_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515be6f38e964ac4899c9224a0db3430" is_pre_election: true
I20260923 17:28:16.544872 10356 raft_consensus.cc:2454] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c1c87fb81f94dd88b1acba0e4dd325a in term 0.
I20260923 17:28:16.544898 10500 raft_consensus.cc:2454] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c1c87fb81f94dd88b1acba0e4dd325a in term 0.
I20260923 17:28:16.545399 10147 leader_election.cc:302] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515be6f38e964ac4899c9224a0db3430, 7c1c87fb81f94dd88b1acba0e4dd325a; no voters:
I20260923 17:28:16.545717 10616 raft_consensus.cc:2781] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:16.545835 10616 raft_consensus.cc:482] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:16.545940 10616 raft_consensus.cc:3037] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:16.548434 10616 raft_consensus.cc:504] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.549558 10356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1621e4dd8d814b3292e91eba6d631222" candidate_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d8695dfb84605a87db878f5708404"
I20260923 17:28:16.549584 10500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1621e4dd8d814b3292e91eba6d631222" candidate_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515be6f38e964ac4899c9224a0db3430"
I20260923 17:28:16.549800 10500 raft_consensus.cc:3037] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:16.549813 10356 raft_consensus.cc:3037] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:16.552870 10356 raft_consensus.cc:2454] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c1c87fb81f94dd88b1acba0e4dd325a in term 1.
I20260923 17:28:16.552857 10500 raft_consensus.cc:2454] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c1c87fb81f94dd88b1acba0e4dd325a in term 1.
I20260923 17:28:16.553457 10147 leader_election.cc:302] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 506d8695dfb84605a87db878f5708404, 515be6f38e964ac4899c9224a0db3430, 7c1c87fb81f94dd88b1acba0e4dd325a; no voters:
I20260923 17:28:16.553766 10608 raft_consensus.cc:2781] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:16.553783 10616 leader_election.cc:288] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [CANDIDATE]: Term 1 election: Requested vote from peers 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835), 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927)
I20260923 17:28:16.553993 10608 raft_consensus.cc:686] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 LEADER]: Becoming Leader. State: Replica: 7c1c87fb81f94dd88b1acba0e4dd325a, State: Running, Role: LEADER
I20260923 17:28:16.554271 10608 consensus_queue.cc:237] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:16.557225 10057 catalog_manager.cc:5495] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193). New cstate: current_term: 1 leader_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } health_report { overall_health: UNKNOWN } } }
I20260923 17:28:16.608338 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.602766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42308 (local address 127.8.29.193:40757)
0923 17:28:16.602934 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.602940 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.603200 (+ 260us) server_negotiation.cc:407] Connection header received
0923 17:28:16.603426 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.603432 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.603516 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.603590 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:16.603978 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.604941 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.606377 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.606897 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.607015 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.607540 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.607679 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.607773 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.607947 (+ 174us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.608034 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:28:16.622531 10614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.616976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.8.29.193:50879)
0923 17:28:16.617177 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:16.617197 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:16.617301 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:16.617650 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:16.617653 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:16.617662 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:28:16.617868 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.617877 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.619272 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.619276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.620263 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:16.620275 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.621292 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.621297 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:16.621445 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.621486 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.621960 (+ 474us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.621973 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 17:28:16.622050 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:28:16.622192 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":1}
I20260923 17:28:16.622570 10563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:16.616995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:50879 (local address 127.8.29.193:40757)
0923 17:28:16.617193 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:16.617199 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:16.617280 (+ 81us) server_negotiation.cc:407] Connection header received
0923 17:28:16.617411 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:16.617416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:16.617514 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:16.617621 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:16.618005 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.619145 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.620462 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:16.621094 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:16.621233 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:16.621666 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:16.621820 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:16.621913 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:16.622100 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 17:28:16.622219 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":83}
I20260923 17:28:16.631960 10356 raft_consensus.cc:1261] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Refusing update from remote peer 7c1c87fb81f94dd88b1acba0e4dd325a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:16.632532 10500 raft_consensus.cc:1261] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Refusing update from remote peer 7c1c87fb81f94dd88b1acba0e4dd325a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:16.632557 10608 consensus_queue.cc:1035] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:16.633302 10608 consensus_queue.cc:1035] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [LEADER]: Connected to new peer: Peer: permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 17:28:16.660840 10595 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:28:18.262773 10057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:18.263007 10057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60738:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:18.267307 10057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a: Sending DeleteTablet for 3 replicas of tablet 1621e4dd8d814b3292e91eba6d631222
I20260923 17:28:18.268275 10336 tablet_service.cc:1514] Processing DeleteTablet for tablet 1621e4dd8d814b3292e91eba6d631222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:18 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:56486
I20260923 17:28:18.268375 10475 tablet_service.cc:1514] Processing DeleteTablet for tablet 1621e4dd8d814b3292e91eba6d631222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:18 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:45388
I20260923 17:28:18.268728 10197 tablet_service.cc:1514] Processing DeleteTablet for tablet 1621e4dd8d814b3292e91eba6d631222 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:18 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:42288
I20260923 17:28:18.269313 10652 tablet_replica.cc:330] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: stopping tablet replica
I20260923 17:28:18.269562 10653 tablet_replica.cc:330] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: stopping tablet replica
I20260923 17:28:18.269580 10652 raft_consensus.cc:2229] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:18.269758 10653 raft_consensus.cc:2229] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:18.269979 10652 raft_consensus.cc:2258] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:18.270303 10653 raft_consensus.cc:2258] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:18.271247 10652 ts_tablet_manager.cc:1905] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:18.276296 10653 ts_tablet_manager.cc:1905] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:18.278861 10652 ts_tablet_manager.cc:1918] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:28:18.279137 10652 log.cc:1196] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/1621e4dd8d814b3292e91eba6d631222
I20260923 17:28:18.279623 10652 ts_tablet_manager.cc:1939] T 1621e4dd8d814b3292e91eba6d631222 P 515be6f38e964ac4899c9224a0db3430: Deleting consensus metadata
I20260923 17:28:18.281143 10044 catalog_manager.cc:4841] TS 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927): tablet 1621e4dd8d814b3292e91eba6d631222 (table test-table [id=dd6bab960f6f4fe98c197ba8992b8700]) successfully deleted
I20260923 17:28:18.282912 10651 tablet_replica.cc:330] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: stopping tablet replica
I20260923 17:28:18.283139 10651 raft_consensus.cc:2229] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:18.283367 10651 raft_consensus.cc:2258] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:18.284054 10653 ts_tablet_manager.cc:1918] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:28:18.284204 10653 log.cc:1196] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/1621e4dd8d814b3292e91eba6d631222
I20260923 17:28:18.284392 10651 ts_tablet_manager.cc:1905] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:18.284654 10653 ts_tablet_manager.cc:1939] T 1621e4dd8d814b3292e91eba6d631222 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting consensus metadata
I20260923 17:28:18.285934 10047 catalog_manager.cc:4841] TS 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757): tablet 1621e4dd8d814b3292e91eba6d631222 (table test-table [id=dd6bab960f6f4fe98c197ba8992b8700]) successfully deleted
I20260923 17:28:18.289950 10651 ts_tablet_manager.cc:1918] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:28:18.290100 10651 log.cc:1196] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/1621e4dd8d814b3292e91eba6d631222
I20260923 17:28:18.290437 10651 ts_tablet_manager.cc:1939] T 1621e4dd8d814b3292e91eba6d631222 P 506d8695dfb84605a87db878f5708404: Deleting consensus metadata
I20260923 17:28:18.291575 10047 catalog_manager.cc:4841] TS 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835): tablet 1621e4dd8d814b3292e91eba6d631222 (table test-table [id=dd6bab960f6f4fe98c197ba8992b8700]) successfully deleted
Sep 23 17:28:18 dist-test-slave-p7jt krb5kdc[10015](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:28:18 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:18.322120 10661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:18.306421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.8.29.254:42027)
0923 17:28:18.306993 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:18.306999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:18.307030 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:28:18.307262 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:18.307266 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:18.307331 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:18.307422 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:18.308836 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.309617 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.310965 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.311477 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.311608 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:18.316646 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:18.316653 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:18.316656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:18.316696 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:18.319856 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:18.320471 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:18.320475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:18.320477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:18.320603 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:18.320900 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:18.320903 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:18.320905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:18.321166 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:18.321352 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:18.321578 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 17:28:18.321766 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":252,"threads_started":1}
I20260923 17:28:18.325526 10057 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:18.326138 10057 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:18.332755 10336 tablet_service.cc:1467] Processing CreateTablet for tablet 900f055325024b72999135ed93783b93 (DEFAULT_TABLE table=test-table [id=17694f8f725b4bd6ba3e4e3d8b079087]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:18.332780 10197 tablet_service.cc:1467] Processing CreateTablet for tablet 900f055325024b72999135ed93783b93 (DEFAULT_TABLE table=test-table [id=17694f8f725b4bd6ba3e4e3d8b079087]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:18.333168 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900f055325024b72999135ed93783b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:18.333195 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900f055325024b72999135ed93783b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:18.333595 10475 tablet_service.cc:1467] Processing CreateTablet for tablet 900f055325024b72999135ed93783b93 (DEFAULT_TABLE table=test-table [id=17694f8f725b4bd6ba3e4e3d8b079087]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:18.333997 10475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 900f055325024b72999135ed93783b93. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:18.339663 10662 tablet_bootstrap.cc:492] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: Bootstrap starting.
I20260923 17:28:18.340525 10664 tablet_bootstrap.cc:492] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: Bootstrap starting.
I20260923 17:28:18.341226 10663 tablet_bootstrap.cc:492] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: Bootstrap starting.
I20260923 17:28:18.341632 10662 tablet_bootstrap.cc:654] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:18.342110 10664 tablet_bootstrap.cc:654] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:18.342725 10663 tablet_bootstrap.cc:654] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:18.344506 10662 tablet_bootstrap.cc:492] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: No bootstrap required, opened a new log
I20260923 17:28:18.344667 10662 ts_tablet_manager.cc:1397] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:28:18.345503 10662 raft_consensus.cc:348] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.345722 10662 raft_consensus.cc:374] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:18.345808 10662 raft_consensus.cc:729] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506d8695dfb84605a87db878f5708404, State: Initialized, Role: FOLLOWER
I20260923 17:28:18.345818 10663 tablet_bootstrap.cc:492] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: No bootstrap required, opened a new log
I20260923 17:28:18.346066 10663 ts_tablet_manager.cc:1397] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260923 17:28:18.346072 10662 consensus_queue.cc:260] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.346673 10664 tablet_bootstrap.cc:492] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: No bootstrap required, opened a new log
I20260923 17:28:18.346823 10664 ts_tablet_manager.cc:1397] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:28:18.347065 10663 raft_consensus.cc:348] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.347280 10663 raft_consensus.cc:374] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:18.347368 10663 raft_consensus.cc:729] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1c87fb81f94dd88b1acba0e4dd325a, State: Initialized, Role: FOLLOWER
I20260923 17:28:18.347610 10663 consensus_queue.cc:260] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.347654 10664 raft_consensus.cc:348] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.347918 10664 raft_consensus.cc:374] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:18.347996 10664 raft_consensus.cc:729] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515be6f38e964ac4899c9224a0db3430, State: Initialized, Role: FOLLOWER
I20260923 17:28:18.348210 10663 ts_tablet_manager.cc:1428] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:18.348253 10664 consensus_queue.cc:260] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.349129 10664 ts_tablet_manager.cc:1428] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:18.349993 10662 ts_tablet_manager.cc:1428] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20260923 17:28:18.380622 10668 raft_consensus.cc:482] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:18.380898 10668 raft_consensus.cc:504] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.381626 10668 leader_election.cc:288] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757), 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927)
I20260923 17:28:18.382200 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900f055325024b72999135ed93783b93" candidate_uuid: "506d8695dfb84605a87db878f5708404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" is_pre_election: true
I20260923 17:28:18.382303 10500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900f055325024b72999135ed93783b93" candidate_uuid: "506d8695dfb84605a87db878f5708404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515be6f38e964ac4899c9224a0db3430" is_pre_election: true
I20260923 17:28:18.382498 10217 raft_consensus.cc:2454] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506d8695dfb84605a87db878f5708404 in term 0.
I20260923 17:28:18.382555 10500 raft_consensus.cc:2454] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 506d8695dfb84605a87db878f5708404 in term 0.
I20260923 17:28:18.383008 10287 leader_election.cc:302] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 506d8695dfb84605a87db878f5708404, 515be6f38e964ac4899c9224a0db3430, 7c1c87fb81f94dd88b1acba0e4dd325a; no voters:
I20260923 17:28:18.383337 10668 raft_consensus.cc:2781] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:18.383458 10668 raft_consensus.cc:482] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:18.383548 10668 raft_consensus.cc:3037] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:18.385695 10668 raft_consensus.cc:504] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.386409 10668 leader_election.cc:288] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [CANDIDATE]: Term 1 election: Requested vote from peers 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757), 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927)
I20260923 17:28:18.386926 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900f055325024b72999135ed93783b93" candidate_uuid: "506d8695dfb84605a87db878f5708404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a"
I20260923 17:28:18.387017 10500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "900f055325024b72999135ed93783b93" candidate_uuid: "506d8695dfb84605a87db878f5708404" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515be6f38e964ac4899c9224a0db3430"
I20260923 17:28:18.387123 10217 raft_consensus.cc:3037] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:18.387233 10500 raft_consensus.cc:3037] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:18.389200 10217 raft_consensus.cc:2454] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506d8695dfb84605a87db878f5708404 in term 1.
I20260923 17:28:18.389256 10500 raft_consensus.cc:2454] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 506d8695dfb84605a87db878f5708404 in term 1.
I20260923 17:28:18.389668 10290 leader_election.cc:302] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 506d8695dfb84605a87db878f5708404, 7c1c87fb81f94dd88b1acba0e4dd325a; no voters:
I20260923 17:28:18.390002 10668 raft_consensus.cc:2781] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:18.390429 10668 raft_consensus.cc:686] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 1 LEADER]: Becoming Leader. State: Replica: 506d8695dfb84605a87db878f5708404, State: Running, Role: LEADER
I20260923 17:28:18.391058 10668 consensus_queue.cc:237] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.394806 10057 catalog_manager.cc:5495] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 reported cstate change: term changed from 0 to 1, leader changed from <none> to 506d8695dfb84605a87db878f5708404 (127.8.29.194). New cstate: current_term: 1 leader_uuid: "506d8695dfb84605a87db878f5708404" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } health_report { overall_health: UNKNOWN } } }
I20260923 17:28:18.417413 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:18.411715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37836 (local address 127.8.29.194:45835)
0923 17:28:18.412239 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:18.412244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:18.412261 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:28:18.412339 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:18.412343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:18.412416 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:18.412581 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:18.412966 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.413753 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.415152 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.415590 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.415699 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:18.416171 (+ 472us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:18.416790 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:18.416925 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:18.417039 (+ 114us) server_negotiation.cc:299] Negotiation successful
0923 17:28:18.417122 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":271,"threads_started":1}
I20260923 17:28:18.436563 10676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:18.427569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:51347 (local address 127.8.29.193:40757)
0923 17:28:18.428117 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:18.428121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:18.431341 (+ 3220us) server_negotiation.cc:407] Connection header received
0923 17:28:18.431545 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:18.431549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:18.431618 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:18.431707 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:18.432209 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.433013 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.434688 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.435153 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.435265 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:18.435748 (+ 483us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:18.435882 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:18.435967 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:18.436109 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 17:28:18.436186 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":261,"threads_started":1}
I20260923 17:28:18.436584 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:18.427491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.8.29.194:51347)
0923 17:28:18.431219 (+ 3728us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:18.431235 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:18.431392 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:18.431745 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:18.431749 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:18.431762 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:28:18.432050 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:18.432059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.433152 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.433156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:18.434543 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:18.434558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.435328 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.435335 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:18.435479 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:18.435541 (+ 62us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:28:18.435989 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:28:18.436001 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 17:28:18.436070 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 17:28:18.436209 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3576,"thread_start_us":3408,"threads_started":1}
I20260923 17:28:18.444972 10500 raft_consensus.cc:1261] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Refusing update from remote peer 506d8695dfb84605a87db878f5708404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:18.445011 10217 raft_consensus.cc:1261] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Refusing update from remote peer 506d8695dfb84605a87db878f5708404: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:18.445585 10668 consensus_queue.cc:1035] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:18.445873 10670 consensus_queue.cc:1035] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:18.487867 10673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:18.481996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:53579 (local address 127.8.29.194:45835)
0923 17:28:18.482203 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:18.482211 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:18.482665 (+ 454us) server_negotiation.cc:407] Connection header received
0923 17:28:18.482839 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:18.482845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:18.482933 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:18.483031 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:18.483565 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.484484 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.486112 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.486591 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.486705 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:18.487143 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:18.487259 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:18.487334 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:18.487500 (+ 166us) server_negotiation.cc:299] Negotiation successful
0923 17:28:18.487579 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:28:18.487933 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:18.481861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:45835 (local address 127.8.29.193:53579)
0923 17:28:18.482578 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:18.482597 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:18.482734 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:18.483064 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:18.483069 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:18.483082 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:28:18.483373 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:18.483383 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.484740 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.484744 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:18.485945 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:18.485971 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.486780 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.486787 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:18.486921 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:18.486971 (+ 50us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:28:18.487369 (+ 398us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:28:18.487380 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:28:18.487451 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:28:18.487600 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":387,"threads_started":1}
I20260923 17:28:18.535856 10059 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:18.536119 10059 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:18.539620 10059 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a: Sending DeleteTablet for 3 replicas of tablet 900f055325024b72999135ed93783b93
I20260923 17:28:18.540560 10336 tablet_service.cc:1514] Processing DeleteTablet for tablet 900f055325024b72999135ed93783b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:18 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:56486
I20260923 17:28:18.540659 10197 tablet_service.cc:1514] Processing DeleteTablet for tablet 900f055325024b72999135ed93783b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:18 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:42288
I20260923 17:28:18.540884 10651 tablet_replica.cc:330] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: stopping tablet replica
I20260923 17:28:18.540918 10653 tablet_replica.cc:330] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: stopping tablet replica
I20260923 17:28:18.540911 10475 tablet_service.cc:1514] Processing DeleteTablet for tablet 900f055325024b72999135ed93783b93 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:18 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:45388
I20260923 17:28:18.541091 10651 raft_consensus.cc:2229] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:18.541097 10653 raft_consensus.cc:2229] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:18.541193 10652 tablet_replica.cc:330] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: stopping tablet replica
I20260923 17:28:18.541285 10653 raft_consensus.cc:2258] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:18.541379 10652 raft_consensus.cc:2229] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:18.541574 10652 raft_consensus.cc:2258] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:18.541683 10651 raft_consensus.cc:2258] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:18.542701 10653 ts_tablet_manager.cc:1905] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:18.543093 10651 ts_tablet_manager.cc:1905] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:18.544018 10059 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:18.544715 10059 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:18.547143 10652 ts_tablet_manager.cc:1905] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:18.549711 10653 ts_tablet_manager.cc:1918] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:28:18.549857 10653 log.cc:1196] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/900f055325024b72999135ed93783b93
I20260923 17:28:18.550253 10653 ts_tablet_manager.cc:1939] T 900f055325024b72999135ed93783b93 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting consensus metadata
I20260923 17:28:18.551904 10047 catalog_manager.cc:4841] TS 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757): tablet 900f055325024b72999135ed93783b93 (table test-table [id=17694f8f725b4bd6ba3e4e3d8b079087]) successfully deleted
I20260923 17:28:18.553040 10336 tablet_service.cc:1467] Processing CreateTablet for tablet 528779b977aa4b5091af2ac824a86778 (DEFAULT_TABLE table=test-table [id=54c63c2c6372450ea173dd2eba876dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:18.553479 10336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 528779b977aa4b5091af2ac824a86778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:18.553416 10197 tablet_service.cc:1467] Processing CreateTablet for tablet 528779b977aa4b5091af2ac824a86778 (DEFAULT_TABLE table=test-table [id=54c63c2c6372450ea173dd2eba876dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:18.553879 10197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 528779b977aa4b5091af2ac824a86778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:18.558949 10662 tablet_bootstrap.cc:492] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: Bootstrap starting.
I20260923 17:28:18.559648 10475 tablet_service.cc:1467] Processing CreateTablet for tablet 528779b977aa4b5091af2ac824a86778 (DEFAULT_TABLE table=test-table [id=54c63c2c6372450ea173dd2eba876dd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:18.560189 10475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 528779b977aa4b5091af2ac824a86778. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:18.560882 10662 tablet_bootstrap.cc:654] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:18.563467 10651 ts_tablet_manager.cc:1918] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:28:18.563637 10651 log.cc:1196] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/900f055325024b72999135ed93783b93
I20260923 17:28:18.564124 10651 ts_tablet_manager.cc:1939] T 900f055325024b72999135ed93783b93 P 506d8695dfb84605a87db878f5708404: Deleting consensus metadata
I20260923 17:28:18.565342 10047 catalog_manager.cc:4841] TS 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835): tablet 900f055325024b72999135ed93783b93 (table test-table [id=17694f8f725b4bd6ba3e4e3d8b079087]) successfully deleted
I20260923 17:28:18.569164 10663 tablet_bootstrap.cc:492] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a: Bootstrap starting.
I20260923 17:28:18.569832 10652 ts_tablet_manager.cc:1918] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:28:18.570030 10652 log.cc:1196] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/900f055325024b72999135ed93783b93
I20260923 17:28:18.570518 10652 ts_tablet_manager.cc:1939] T 900f055325024b72999135ed93783b93 P 515be6f38e964ac4899c9224a0db3430: Deleting consensus metadata
I20260923 17:28:18.571244 10663 tablet_bootstrap.cc:654] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:18.571945 10044 catalog_manager.cc:4841] TS 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927): tablet 900f055325024b72999135ed93783b93 (table test-table [id=17694f8f725b4bd6ba3e4e3d8b079087]) successfully deleted
I20260923 17:28:18.572615 10664 tablet_bootstrap.cc:492] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: Bootstrap starting.
I20260923 17:28:18.574299 10664 tablet_bootstrap.cc:654] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:18.575052 10662 tablet_bootstrap.cc:492] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: No bootstrap required, opened a new log
I20260923 17:28:18.575273 10662 ts_tablet_manager.cc:1397] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.003s
I20260923 17:28:18.576233 10662 raft_consensus.cc:348] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.576584 10662 raft_consensus.cc:374] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:18.576735 10662 raft_consensus.cc:729] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 506d8695dfb84605a87db878f5708404, State: Initialized, Role: FOLLOWER
I20260923 17:28:18.577263 10662 consensus_queue.cc:260] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.578106 10662 ts_tablet_manager.cc:1428] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:18.580217 10664 tablet_bootstrap.cc:492] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: No bootstrap required, opened a new log
I20260923 17:28:18.580543 10663 tablet_bootstrap.cc:492] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a: No bootstrap required, opened a new log
I20260923 17:28:18.580782 10663 ts_tablet_manager.cc:1397] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260923 17:28:18.580590 10664 ts_tablet_manager.cc:1397] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:28:18.581691 10664 raft_consensus.cc:348] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.581664 10663 raft_consensus.cc:348] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.581933 10664 raft_consensus.cc:374] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:18.582000 10663 raft_consensus.cc:374] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:18.582015 10664 raft_consensus.cc:729] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515be6f38e964ac4899c9224a0db3430, State: Initialized, Role: FOLLOWER
I20260923 17:28:18.582082 10663 raft_consensus.cc:729] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1c87fb81f94dd88b1acba0e4dd325a, State: Initialized, Role: FOLLOWER
I20260923 17:28:18.582307 10663 consensus_queue.cc:260] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.582307 10664 consensus_queue.cc:260] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.582894 10663 ts_tablet_manager.cc:1428] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:18.586001 10664 ts_tablet_manager.cc:1428] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260923 17:28:18.617380 10669 raft_consensus.cc:482] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:18.617650 10669 raft_consensus.cc:504] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.618395 10669 leader_election.cc:288] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757), 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835)
I20260923 17:28:18.619037 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "528779b977aa4b5091af2ac824a86778" candidate_uuid: "515be6f38e964ac4899c9224a0db3430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" is_pre_election: true
I20260923 17:28:18.619086 10356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "528779b977aa4b5091af2ac824a86778" candidate_uuid: "515be6f38e964ac4899c9224a0db3430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d8695dfb84605a87db878f5708404" is_pre_election: true
I20260923 17:28:18.619289 10217 raft_consensus.cc:2454] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 515be6f38e964ac4899c9224a0db3430 in term 0.
I20260923 17:28:18.619370 10356 raft_consensus.cc:2454] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 515be6f38e964ac4899c9224a0db3430 in term 0.
I20260923 17:28:18.619741 10429 leader_election.cc:302] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515be6f38e964ac4899c9224a0db3430, 7c1c87fb81f94dd88b1acba0e4dd325a; no voters:
I20260923 17:28:18.620000 10669 raft_consensus.cc:2781] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:18.620117 10669 raft_consensus.cc:482] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:18.620206 10669 raft_consensus.cc:3037] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:18.623221 10669 raft_consensus.cc:504] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.623883 10669 leader_election.cc:288] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [CANDIDATE]: Term 1 election: Requested vote from peers 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757), 506d8695dfb84605a87db878f5708404 (127.8.29.194:45835)
I20260923 17:28:18.624456 10217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "528779b977aa4b5091af2ac824a86778" candidate_uuid: "515be6f38e964ac4899c9224a0db3430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a"
I20260923 17:28:18.624521 10356 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "528779b977aa4b5091af2ac824a86778" candidate_uuid: "515be6f38e964ac4899c9224a0db3430" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "506d8695dfb84605a87db878f5708404"
I20260923 17:28:18.624703 10217 raft_consensus.cc:3037] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:18.624763 10356 raft_consensus.cc:3037] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:18.627688 10217 raft_consensus.cc:2454] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 515be6f38e964ac4899c9224a0db3430 in term 1.
I20260923 17:28:18.627696 10356 raft_consensus.cc:2454] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 515be6f38e964ac4899c9224a0db3430 in term 1.
I20260923 17:28:18.628252 10429 leader_election.cc:302] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 515be6f38e964ac4899c9224a0db3430, 7c1c87fb81f94dd88b1acba0e4dd325a; no voters:
I20260923 17:28:18.628576 10669 raft_consensus.cc:2781] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:18.629184 10669 raft_consensus.cc:686] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 1 LEADER]: Becoming Leader. State: Replica: 515be6f38e964ac4899c9224a0db3430, State: Running, Role: LEADER
I20260923 17:28:18.629490 10669 consensus_queue.cc:237] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } }
I20260923 17:28:18.632536 10057 catalog_manager.cc:5495] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 reported cstate change: term changed from 0 to 1, leader changed from <none> to 515be6f38e964ac4899c9224a0db3430 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "515be6f38e964ac4899c9224a0db3430" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "515be6f38e964ac4899c9224a0db3430" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33927 } health_report { overall_health: HEALTHY } } }
I20260923 17:28:18.673309 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:18.655950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46326 (local address 127.8.29.195:33927)
0923 17:28:18.668324 (+ 12374us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:18.668329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:18.668357 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:28:18.668433 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:18.668437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:18.668511 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:18.668640 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:18.669332 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.670130 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.671516 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:18.671934 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:18.672049 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:18.672527 (+ 478us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:18.672662 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:18.672739 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:18.672889 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 17:28:18.672970 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12210,"thread_start_us":12856,"threads_started":1}
I20260923 17:28:18.684059 10356 raft_consensus.cc:1261] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Refusing update from remote peer 515be6f38e964ac4899c9224a0db3430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:18.684223 10217 raft_consensus.cc:1261] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Refusing update from remote peer 515be6f38e964ac4899c9224a0db3430: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:18.684762 10688 consensus_queue.cc:1035] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c1c87fb81f94dd88b1acba0e4dd325a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 40757 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:18.685087 10669 consensus_queue.cc:1035] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [LEADER]: Connected to new peer: Peer: permanent_uuid: "506d8695dfb84605a87db878f5708404" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 45835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:18.691649 10634 mvcc.cc:205] Tried to move back new op lower bound from 7332595706604642304 to 7332595706388807680. Current Snapshot: MvccSnapshot[applied={T|T < 7332595706604642304}]
W20260923 17:28:18.709946 10634 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 17:28:20.301162 10057 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:20.301422 10057 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:20.305814 10057 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abd24a37b98c412dbe7304c850fe199a: Sending DeleteTablet for 3 replicas of tablet 528779b977aa4b5091af2ac824a86778
I20260923 17:28:20.306820 10336 tablet_service.cc:1514] Processing DeleteTablet for tablet 528779b977aa4b5091af2ac824a86778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:20 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:56486
I20260923 17:28:20.306802 10197 tablet_service.cc:1514] Processing DeleteTablet for tablet 528779b977aa4b5091af2ac824a86778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:20 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:42288
I20260923 17:28:20.306895 10475 tablet_service.cc:1514] Processing DeleteTablet for tablet 528779b977aa4b5091af2ac824a86778 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:20 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:45388
I20260923 17:28:20.307436 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10131
I20260923 17:28:20.318593 10718 tablet_replica.cc:330] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: stopping tablet replica
I20260923 17:28:20.318850 10718 raft_consensus.cc:2229] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:20.319142 10718 raft_consensus.cc:2258] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:20.319644 10719 tablet_replica.cc:330] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: stopping tablet replica
I20260923 17:28:20.319932 10719 raft_consensus.cc:2229] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:20.320389 10719 raft_consensus.cc:2258] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:20.320456 10718 ts_tablet_manager.cc:1905] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:20.321907 10719 ts_tablet_manager.cc:1905] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:20.324081 10717 tablet_replica.cc:330] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a: stopping tablet replica
I20260923 17:28:20.324319 10717 raft_consensus.cc:2229] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:20.324602 10717 raft_consensus.cc:2258] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:20.325596 10717 ts_tablet_manager.cc:1905] T 528779b977aa4b5091af2ac824a86778 P 7c1c87fb81f94dd88b1acba0e4dd325a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 17:28:20 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:20.343429 10721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.308976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.8.29.193:40757)
0923 17:28:20.324307 (+ 15331us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:20.324313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:20.324346 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:28:20.324417 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:20.324422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:20.324522 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:20.324657 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:20.326021 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:20.326889 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:20.328264 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:20.328685 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:20.328850 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:20.337249 (+ 8399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:20.337267 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:20.337284 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:20.337326 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:20.340324 (+ 2998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:20.341133 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:20.341142 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:20.341144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:20.341298 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:20.341741 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:20.341745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:20.341748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:20.342148 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:20.342443 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:20.342835 (+ 392us) server_negotiation.cc:299] Negotiation successful
0923 17:28:20.343085 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15176,"thread_start_us":15411,"threads_started":1}
W20260923 17:28:20.344123 10262 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59980
W20260923 17:28:20.344875 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10131: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:28:20.345127 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 10131
I20260923 17:28:20.367062 10719 ts_tablet_manager.cc:1918] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:28:20.367062 10718 ts_tablet_manager.cc:1918] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 17:28:20.367230 10718 log.cc:1196] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/528779b977aa4b5091af2ac824a86778
I20260923 17:28:20.367405 10719 log.cc:1196] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/528779b977aa4b5091af2ac824a86778
I20260923 17:28:20.367671 10718 ts_tablet_manager.cc:1939] T 528779b977aa4b5091af2ac824a86778 P 506d8695dfb84605a87db878f5708404: Deleting consensus metadata
I20260923 17:28:20.368001 10719 ts_tablet_manager.cc:1939] T 528779b977aa4b5091af2ac824a86778 P 515be6f38e964ac4899c9224a0db3430: Deleting consensus metadata
I20260923 17:28:20.385056 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10271
W20260923 17:28:20.385183 10047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.29.193:40757 (error 108)
I20260923 17:28:20.387339 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.386435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.0.0.1:59986)
0923 17:28:20.386963 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:20.387099 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:40757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":230,"threads_started":1}
W20260923 17:28:20.387532 10047 catalog_manager.cc:4584] TS 7c1c87fb81f94dd88b1acba0e4dd325a (127.8.29.193:40757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 528779b977aa4b5091af2ac824a86778: Network error: Client connection negotiation failed: client connection to 127.8.29.193:40757: connect: Connection refused (error 111)
Sep 23 17:28:20 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:28:20.400487 10723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.385797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.8.29.194:45835)
0923 17:28:20.386319 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:20.386323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:20.386338 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:28:20.386391 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:20.386395 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:20.386481 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:20.386579 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:20.387982 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:20.388750 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:20.390120 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:20.390691 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:20.390800 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:20.395041 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:20.395055 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:20.395074 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:20.395118 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:20.397931 (+ 2813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:20.398678 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:20.398687 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:20.398689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:20.398827 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:20.399184 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:20.399188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:20.399190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:20.399561 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:20.399792 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:20.400057 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 17:28:20.400234 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":297,"threads_started":1}
W20260923 17:28:20.401268 10401 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37842
W20260923 17:28:20.402107 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10271: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:28:20.402266 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 10271
I20260923 17:28:20.426750 10044 catalog_manager.cc:4841] TS 515be6f38e964ac4899c9224a0db3430 (127.8.29.195:33927): tablet 528779b977aa4b5091af2ac824a86778 (table test-table [id=54c63c2c6372450ea173dd2eba876dd0]) successfully deleted
I20260923 17:28:20.441139 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.440622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.0.0.1:59998)
0923 17:28:20.440879 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:20.440981 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:40757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260923 17:28:20.442225 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.441857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.0.0.1:60002)
0923 17:28:20.442009 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:20.442088 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:40757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 17:28:20.443270 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10410
I20260923 17:28:20.444015 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.443596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:45835 (local address 127.0.0.1:37850)
0923 17:28:20.443753 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:20.443854 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.194:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
Sep 23 17:28:20 dist-test-slave-p7jt krb5kdc[10015](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184498, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:28:20.459080 10725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.444033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.8.29.195:33927)
0923 17:28:20.444479 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:20.444483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:20.444516 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:28:20.444572 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:20.444577 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:20.444663 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:20.444758 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:20.445962 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:20.446794 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:20.448201 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:20.448618 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:20.448725 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:20.453027 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:20.453044 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:20.453061 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:20.453100 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:20.456234 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:20.456976 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:20.456987 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:20.456991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:20.457167 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:20.457482 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:20.457485 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:20.457487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:20.457870 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:20.458083 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:20.458557 (+ 474us) server_negotiation.cc:299] Negotiation successful
0923 17:28:20.458763 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":225,"threads_started":1}
W20260923 17:28:20.459865 10540 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46330
W20260923 17:28:20.460580 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10410: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:28:20.460743 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 10410
I20260923 17:28:20.476799 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.476245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:45835 (local address 127.0.0.1:37864)
0923 17:28:20.476527 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:20.476623 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.194:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260923 17:28:20.477946 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.477535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:45835 (local address 127.0.0.1:37874)
0923 17:28:20.477746 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:20.477824 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.194:45835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 17:28:20.479246 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10028
I20260923 17:28:20.480297 10100 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:20.602336 10724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:20.601836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:40757 (local address 127.0.0.1:60010)
0923 17:28:20.602018 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:20.602130 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:40757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:28:20.602541 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 10028
2026-09-23T17:28:20Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9407 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:21 dist-test-slave-p7jt krb5kdc[10730](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:21 dist-test-slave-p7jt krb5kdc[10730](info): set up 2 sockets
Sep 23 17:28:21 dist-test-slave-p7jt krb5kdc[10730](info): commencing operation
krb5kdc: starting...
W20260923 17:28:22.298724 8311 mini_kdc.cc:121] Time spent starting KDC: real 1.656s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:22 dist-test-slave-p7jt krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:28:22.485534 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:38623
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:22.813728 10743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:22.814028 10743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:22.814167 10743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:22.821516 10743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:22.821633 10743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:22.821688 10743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:22.821764 10743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:22.821820 10743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:22.831759 10743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:22.833146 10743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:22.834544 10743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:22.846698 10751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:22.846671 10748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:22.846756 10749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:22.852062 10743 server_base.cc:1034] running on GCE node
I20260923 17:28:22.853037 10743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:22.854563 10743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:22.855808 10743 hybrid_clock.cc:648] HybridClock initialized: now 1790184502855759 us; error 33 us; skew 500 ppm
I20260923 17:28:22.856200 10743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:22 dist-test-slave-p7jt krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184502, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:22.859341 10743 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:22.860848 10743 webserver.cc:460] Webserver started at http://127.8.29.254:43763/ using document root <none> and password file <none>
I20260923 17:28:22.861584 10743 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:22.861706 10743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:22.862005 10743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:22.865109 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "6a1dcc94254046eaba8d78f3ca1be776"
format_stamp: "Formatted at 2026-09-23 17:28:22 on dist-test-slave-p7jt"
server_key: "c648b38b104e674320a0e2c6d01c463e"
server_key_iv: "90758f8aa4eecc2b03bffa5f1da179e5"
server_key_version: "encryptionkey@0"
I20260923 17:28:22.865877 10743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "6a1dcc94254046eaba8d78f3ca1be776"
format_stamp: "Formatted at 2026-09-23 17:28:22 on dist-test-slave-p7jt"
server_key: "c648b38b104e674320a0e2c6d01c463e"
server_key_iv: "90758f8aa4eecc2b03bffa5f1da179e5"
server_key_version: "encryptionkey@0"
I20260923 17:28:22.871070 10743 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 17:28:22.874825 10758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:22.876408 10743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:22.876600 10743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "6a1dcc94254046eaba8d78f3ca1be776"
format_stamp: "Formatted at 2026-09-23 17:28:22 on dist-test-slave-p7jt"
server_key: "c648b38b104e674320a0e2c6d01c463e"
server_key_iv: "90758f8aa4eecc2b03bffa5f1da179e5"
server_key_version: "encryptionkey@0"
I20260923 17:28:22.876803 10743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:22.902303 10743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:22.906683 10743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:22.906963 10743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:22.924785 10743 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:38623
I20260923 17:28:22.924795 10809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:38623 every 8 connection(s)
I20260923 17:28:22.926205 10743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:22.930059 10810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:22.936671 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 10743
I20260923 17:28:22.936874 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:22.937522 8311 external_mini_cluster.cc:1428] Setting key ec6299a13a644d690a8ac8ecfa366c14
I20260923 17:28:22.941578 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776: Bootstrap starting.
I20260923 17:28:22.945482 10810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:22.946861 10810 log.cc:824] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:22.949644 10810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776: No bootstrap required, opened a new log
Sep 23 17:28:22 dist-test-slave-p7jt krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184502, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:22.957460 10810 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1dcc94254046eaba8d78f3ca1be776" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38623 } }
I20260923 17:28:22.957886 10810 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:22.957983 10810 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a1dcc94254046eaba8d78f3ca1be776, State: Initialized, Role: FOLLOWER
I20260923 17:28:22.958674 10810 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1dcc94254046eaba8d78f3ca1be776" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38623 } }
I20260923 17:28:22.958921 10810 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:22.959077 10810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:22.959244 10810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:22.961216 10813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:22.939769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47972 (local address 127.8.29.254:38623)
0923 17:28:22.940588 (+ 819us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:22.940604 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:22.940654 (+ 50us) server_negotiation.cc:407] Connection header received
0923 17:28:22.943075 (+ 2421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:22.943107 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:22.943703 (+ 596us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:22.944189 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:22.945488 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:22.947496 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:22.948808 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:22.949477 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:22.949661 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:22.953440 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:22.953448 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:22.953467 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:22.953508 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:22.956713 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:22.957511 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:22.957518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:22.957525 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:22.957708 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:22.958171 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:22.958177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:22.958181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:22.958840 (+ 659us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:22.959149 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:22.959594 (+ 445us) server_negotiation.cc:299] Negotiation successful
0923 17:28:22.960064 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":363,"threads_started":1}
I20260923 17:28:22.962056 10810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1dcc94254046eaba8d78f3ca1be776" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38623 } }
I20260923 17:28:22.962693 10810 leader_election.cc:302] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a1dcc94254046eaba8d78f3ca1be776; no voters:
I20260923 17:28:22.963469 10810 leader_election.cc:288] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:22.963610 10815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:22.964756 10815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [term 1 LEADER]: Becoming Leader. State: Replica: 6a1dcc94254046eaba8d78f3ca1be776, State: Running, Role: LEADER
I20260923 17:28:22.965318 10815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1dcc94254046eaba8d78f3ca1be776" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38623 } }
I20260923 17:28:22.965660 10810 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:22.968135 10816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a1dcc94254046eaba8d78f3ca1be776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1dcc94254046eaba8d78f3ca1be776" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38623 } } }
I20260923 17:28:22.969537 10817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a1dcc94254046eaba8d78f3ca1be776. Latest consensus state: current_term: 1 leader_uuid: "6a1dcc94254046eaba8d78f3ca1be776" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a1dcc94254046eaba8d78f3ca1be776" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38623 } } }
I20260923 17:28:22.970326 10817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:22.970403 10816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776 [sys.catalog]: This master's current role is: LEADER
W20260923 17:28:22.977628 10831 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:28:22.977905 10831 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:28:22.978075 10825 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:22.979821 10825 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:22.988580 10825 catalog_manager.cc:1347] Generated new cluster ID: 0f2cddbe73e64b27b1218c90f3ebb2b7
I20260923 17:28:22.988705 10825 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:23.009402 10825 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:23.010893 10825 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:23.018721 10825 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a1dcc94254046eaba8d78f3ca1be776: Generated new TSK 0
I20260923 17:28:23.019631 10825 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:28:23.072376 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38623
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:23.397413 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:23.397692 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:23.397789 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:23.405133 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:23.405254 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:23.405369 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:23.415380 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:23.416644 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:23.417870 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:23.431020 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:23.431082 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:23.431020 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:23.431932 10836 server_base.cc:1034] running on GCE node
I20260923 17:28:23.432523 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:23.433533 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:23.434793 10836 hybrid_clock.cc:648] HybridClock initialized: now 1790184503434723 us; error 62 us; skew 500 ppm
I20260923 17:28:23.435142 10836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:23 dist-test-slave-p7jt krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184503, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:23.438513 10836 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:28:23.440176 10836 webserver.cc:460] Webserver started at http://127.8.29.193:40075/ using document root <none> and password file <none>
I20260923 17:28:23.440901 10836 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:23.441030 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:23.441339 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:23.444495 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "8b87b1da956c465d946079a3d18ccf89"
format_stamp: "Formatted at 2026-09-23 17:28:23 on dist-test-slave-p7jt"
server_key: "500c43cf5a0c96adf570785221629276"
server_key_iv: "ed5cef01b4f1dd71fd11bb0a32bcb830"
server_key_version: "encryptionkey@0"
I20260923 17:28:23.445281 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "8b87b1da956c465d946079a3d18ccf89"
format_stamp: "Formatted at 2026-09-23 17:28:23 on dist-test-slave-p7jt"
server_key: "500c43cf5a0c96adf570785221629276"
server_key_iv: "ed5cef01b4f1dd71fd11bb0a32bcb830"
server_key_version: "encryptionkey@0"
I20260923 17:28:23.450767 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:28:23.454526 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:23.455935 10836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:23.456146 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "8b87b1da956c465d946079a3d18ccf89"
format_stamp: "Formatted at 2026-09-23 17:28:23 on dist-test-slave-p7jt"
server_key: "500c43cf5a0c96adf570785221629276"
server_key_iv: "ed5cef01b4f1dd71fd11bb0a32bcb830"
server_key_version: "encryptionkey@0"
I20260923 17:28:23.456347 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:23.484915 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:23.489296 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:23.489594 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:23.490674 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:23.492157 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:23.492240 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:23.492363 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:23.492414 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:23.523350 10836 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:43509
I20260923 17:28:23.523413 10963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:43509 every 8 connection(s)
I20260923 17:28:23.524637 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:23.530987 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 10836
I20260923 17:28:23.531177 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:23.531791 8311 external_mini_cluster.cc:1428] Setting key 7a2669e57026bc87df5a52780b48b85c
Sep 23 17:28:23 dist-test-slave-p7jt krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184503, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:23.546137 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.527263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:34529 (local address 127.8.29.254:38623)
0923 17:28:23.527765 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:23.527769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:23.528669 (+ 900us) server_negotiation.cc:407] Connection header received
0923 17:28:23.530562 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:23.530569 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:23.530662 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:23.530758 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:23.532536 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.533574 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.535275 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.535688 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.535799 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:23.539790 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:23.539795 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:23.539798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:23.539833 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:23.542663 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:23.543531 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:23.543536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:23.543538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:23.543677 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:23.544171 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:23.544177 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:23.544179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:23.544515 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:23.544664 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:23.545469 (+ 805us) server_negotiation.cc:299] Negotiation successful
0923 17:28:23.545719 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":287,"threads_started":1}
I20260923 17:28:23.547206 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:23.527717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38623 (local address 127.8.29.193:34529)
0923 17:28:23.528476 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:23.528518 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:23.530247 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:23.531043 (+ 796us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:23.531053 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:23.531664 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:23.532336 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:23.532355 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.533745 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.533749 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:23.535109 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:23.535128 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:23.535936 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:23.535944 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:23.536180 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:23.536955 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:23.536993 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:23.539546 (+ 2553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:23.542838 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:23.542850 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:23.542868 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:23.543371 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:23.543812 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:23.543818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:23.543822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:23.544036 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:23.544662 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:23.544672 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:23.545051 (+ 379us) client_negotiation.cc:769] Sending connection context
0923 17:28:23.545332 (+ 281us) client_negotiation.cc:240] Negotiation successful
0923 17:28:23.545909 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":308,"threads_started":1}
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:28:23.549043 10964 heartbeater.cc:347] Connected to a master server at 127.8.29.254:38623
I20260923 17:28:23.549402 10964 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:23.550200 10964 heartbeater.cc:511] Master 127.8.29.254:38623 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
I20260923 17:28:23.552546 10775 ts_manager.cc:194] Registered new tserver with Master: 8b87b1da956c465d946079a3d18ccf89 (127.8.29.193:43509)
I20260923 17:28:23.554114 10775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:34529
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:28:23.574954 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38623
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:23.895988 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:23.896291 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:23.896412 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:23.903623 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:23.903769 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:23.903957 10970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:23.914060 10970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:23.915366 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:23.916678 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:23.929176 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:23.929134 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:23.929683 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:23.929965 10970 server_base.cc:1034] running on GCE node
I20260923 17:28:23.930657 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:23.931679 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:23.932902 10970 hybrid_clock.cc:648] HybridClock initialized: now 1790184503932852 us; error 46 us; skew 500 ppm
I20260923 17:28:23.933228 10970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:23 dist-test-slave-p7jt krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184503, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:23.936461 10970 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:28:23.937994 10970 webserver.cc:460] Webserver started at http://127.8.29.194:36333/ using document root <none> and password file <none>
I20260923 17:28:23.938727 10970 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:23.938845 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:23.939134 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:23.941941 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "e8579f141746432791b52ceb88617673"
format_stamp: "Formatted at 2026-09-23 17:28:23 on dist-test-slave-p7jt"
server_key: "5ff9e12159d06ca55f68ff9ee3b2deae"
server_key_iv: "043e7fbfb52eebbb5b04004b2d4d395b"
server_key_version: "encryptionkey@0"
I20260923 17:28:23.942747 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "e8579f141746432791b52ceb88617673"
format_stamp: "Formatted at 2026-09-23 17:28:23 on dist-test-slave-p7jt"
server_key: "5ff9e12159d06ca55f68ff9ee3b2deae"
server_key_iv: "043e7fbfb52eebbb5b04004b2d4d395b"
server_key_version: "encryptionkey@0"
I20260923 17:28:23.947997 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:28:23.951578 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:23.952958 10970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:28:23.953102 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "e8579f141746432791b52ceb88617673"
format_stamp: "Formatted at 2026-09-23 17:28:23 on dist-test-slave-p7jt"
server_key: "5ff9e12159d06ca55f68ff9ee3b2deae"
server_key_iv: "043e7fbfb52eebbb5b04004b2d4d395b"
server_key_version: "encryptionkey@0"
I20260923 17:28:23.953287 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:23.977373 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:23.982045 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:23.982326 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:23.983381 10970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:23.984788 10970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:23.984866 10970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:23.984980 10970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:23.985040 10970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:24.015516 10970 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:32931
I20260923 17:28:24.015564 11097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:32931 every 8 connection(s)
I20260923 17:28:24.016773 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:24.023029 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 10970
I20260923 17:28:24.023204 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:24.023689 8311 external_mini_cluster.cc:1428] Setting key 75d3cb0b73fa468f7542d5b4c998f484
Sep 23 17:28:24 dist-test-slave-p7jt krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184503, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:24.038367 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.019390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:60601 (local address 127.8.29.254:38623)
0923 17:28:24.019602 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:24.019608 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:24.020984 (+ 1376us) server_negotiation.cc:407] Connection header received
0923 17:28:24.022843 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:24.022849 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:24.022989 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:24.023117 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:24.024893 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.025734 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.027421 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.027834 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.027943 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:24.031840 (+ 3897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:24.031846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:24.031851 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:24.031891 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:24.035153 (+ 3262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:24.036006 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:24.036012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:24.036017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:24.036170 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:24.036619 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:24.036624 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:24.036627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:24.037000 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:24.037151 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:24.037937 (+ 786us) server_negotiation.cc:299] Negotiation successful
0923 17:28:24.038127 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:28:24.040025 11100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.019938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38623 (local address 127.8.29.194:60601)
0923 17:28:24.020755 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:24.020800 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:24.022516 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:24.023421 (+ 905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:24.023430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:24.024011 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:24.024724 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:24.024743 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.025893 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.025898 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:24.027281 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:24.027294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.028056 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.028064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:24.028283 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:24.028907 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:24.028930 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:24.031575 (+ 2645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:24.035327 (+ 3752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:24.035336 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:24.035352 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:24.035840 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:24.036290 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:24.036294 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:24.036296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:24.036501 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:24.037120 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:24.037127 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:24.037507 (+ 380us) client_negotiation.cc:769] Sending connection context
0923 17:28:24.037830 (+ 323us) client_negotiation.cc:240] Negotiation successful
0923 17:28:24.038435 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":399,"threads_started":1}
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
I20260923 17:28:24.041569 11098 heartbeater.cc:347] Connected to a master server at 127.8.29.254:38623
I20260923 17:28:24.041898 11098 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:24.042703 11098 heartbeater.cc:511] Master 127.8.29.254:38623 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
I20260923 17:28:24.044409 10775 ts_manager.cc:194] Registered new tserver with Master: e8579f141746432791b52ceb88617673 (127.8.29.194:32931)
I20260923 17:28:24.045506 10775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:60601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:28:24.065560 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38623
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:24.388899 11103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:24.389209 11103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:24.389328 11103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:24.396754 11103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:24.396881 11103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:24.397079 11103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:24.407238 11103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:24.408571 11103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:24.409888 11103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:24.423902 11108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:24.423950 11109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:24.424005 11111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:24.424769 11103 server_base.cc:1034] running on GCE node
I20260923 17:28:24.425518 11103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:24.426863 11103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:24.428128 11103 hybrid_clock.cc:648] HybridClock initialized: now 1790184504428050 us; error 67 us; skew 500 ppm
I20260923 17:28:24.428503 11103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:24 dist-test-slave-p7jt krb5kdc[10730](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184504, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:24.431870 11103 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:28:24.433395 11103 webserver.cc:460] Webserver started at http://127.8.29.195:42071/ using document root <none> and password file <none>
I20260923 17:28:24.434104 11103 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:24.434222 11103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:24.434568 11103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:24.437561 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "afcb4051ed224867aa2e1f2b622ba1cf"
format_stamp: "Formatted at 2026-09-23 17:28:24 on dist-test-slave-p7jt"
server_key: "3aa8b39e64dc92dd7aaa5a74e8d90e51"
server_key_iv: "fa479639dc58e2bd961ebddfb11ef4fc"
server_key_version: "encryptionkey@0"
I20260923 17:28:24.438359 11103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "afcb4051ed224867aa2e1f2b622ba1cf"
format_stamp: "Formatted at 2026-09-23 17:28:24 on dist-test-slave-p7jt"
server_key: "3aa8b39e64dc92dd7aaa5a74e8d90e51"
server_key_iv: "fa479639dc58e2bd961ebddfb11ef4fc"
server_key_version: "encryptionkey@0"
I20260923 17:28:24.443841 11103 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:28:24.447692 11118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:24.449281 11103 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:28:24.449461 11103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "afcb4051ed224867aa2e1f2b622ba1cf"
format_stamp: "Formatted at 2026-09-23 17:28:24 on dist-test-slave-p7jt"
server_key: "3aa8b39e64dc92dd7aaa5a74e8d90e51"
server_key_iv: "fa479639dc58e2bd961ebddfb11ef4fc"
server_key_version: "encryptionkey@0"
I20260923 17:28:24.449687 11103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:24.484221 11103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:24.488377 11103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:24.488659 11103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:24.489642 11103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:24.491081 11103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:24.491163 11103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:24.491286 11103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:24.491338 11103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:24.524443 11103 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:38793
I20260923 17:28:24.524520 11230 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:38793 every 8 connection(s)
I20260923 17:28:24.525959 11103 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:24.533793 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 11103
I20260923 17:28:24.533986 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:24.534605 8311 external_mini_cluster.cc:1428] Setting key 108299b44ef6b8f75080705ec2f3247b
Sep 23 17:28:24 dist-test-slave-p7jt krb5kdc[10730](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184504, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:24.546980 10966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.530423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:45507 (local address 127.8.29.254:38623)
0923 17:28:24.530631 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:24.530637 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:24.531166 (+ 529us) server_negotiation.cc:407] Connection header received
0923 17:28:24.533154 (+ 1988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:24.533159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:24.533265 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:24.533362 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:24.535536 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.536417 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.538058 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.538498 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.538591 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:24.541889 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:24.541894 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:24.541897 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:24.541930 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:24.544341 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:24.545009 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:24.545012 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:24.545015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:24.545153 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:24.545540 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:24.545543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:24.545544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:24.545788 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:24.545889 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:24.546555 (+ 666us) server_negotiation.cc:299] Negotiation successful
0923 17:28:24.546725 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:28:24.548146 11233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:24.529992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38623 (local address 127.8.29.195:45507)
0923 17:28:24.530971 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:24.531019 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:24.532864 (+ 1845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:24.533758 (+ 894us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:24.533773 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:24.534598 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:24.535371 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:24.535393 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.536550 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.536554 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:24.537932 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:24.537945 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:24.538696 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:24.538702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:24.538903 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:24.539560 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:24.539596 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:24.541696 (+ 2100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:24.544475 (+ 2779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:24.544482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:24.544497 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:24.544922 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:24.545267 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:24.545271 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:24.545274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:24.545460 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:24.545875 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:24.545881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:24.546211 (+ 330us) client_negotiation.cc:769] Sending connection context
0923 17:28:24.546445 (+ 234us) client_negotiation.cc:240] Negotiation successful
0923 17:28:24.546908 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":679,"thread_start_us":409,"threads_started":1}
I20260923 17:28:24.549690 11231 heartbeater.cc:347] Connected to a master server at 127.8.29.254:38623
I20260923 17:28:24.550052 11231 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:24.550799 11231 heartbeater.cc:511] Master 127.8.29.254:38623 requested a full tablet report, sending...
I20260923 17:28:24.552096 10775 ts_manager.cc:194] Registered new tserver with Master: afcb4051ed224867aa2e1f2b622ba1cf (127.8.29.195:38793)
I20260923 17:28:24.552865 10775 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:45507
I20260923 17:28:24.557335 10964 heartbeater.cc:503] Master 127.8.29.254:38623 was elected leader, sending a full tablet report...
I20260923 17:28:24.562191 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:28:24.591361 10966 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:24.588890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.8.29.254:38623)
0923 17:28:24.589057 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:24.589063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:24.589485 (+ 422us) server_negotiation.cc:407] Connection header received
0923 17:28:24.589664 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:24.589669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:24.589738 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:24.589823 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:24.591112 (+ 1289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47980: BlockingRecv error: recv got EOF from 127.0.0.1:47980 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:28:24.594038 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10836
W20260923 17:28:24.596766 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10836: Not authorized: Client connection negotiation failed: client connection to 127.8.29.193:43509: server requires authentication, but client does not have Kerberos credentials available
I20260923 17:28:24.596939 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 10836
I20260923 17:28:24.609328 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10970
W20260923 17:28:24.612038 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10970: Not authorized: Client connection negotiation failed: client connection to 127.8.29.194:32931: server requires authentication, but client does not have Kerberos credentials available
I20260923 17:28:24.612201 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 10970
I20260923 17:28:24.624373 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11103
W20260923 17:28:24.627246 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11103: Not authorized: Client connection negotiation failed: client connection to 127.8.29.195:38793: server requires authentication, but client does not have Kerberos credentials available
I20260923 17:28:24.627388 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 11103
I20260923 17:28:24.639370 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 10743
I20260923 17:28:24.640578 10805 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:24.744292 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 10743
2026-09-23T17:28:24Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4139 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:24 dist-test-slave-p7jt krb5kdc[11248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:24 dist-test-slave-p7jt krb5kdc[11248](info): set up 2 sockets
Sep 23 17:28:24 dist-test-slave-p7jt krb5kdc[11248](info): commencing operation
krb5kdc: starting...
W20260923 17:28:25.754673 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.973s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:25 dist-test-slave-p7jt krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:28:25.942435 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40093
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:44661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:40093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:26.258356 11261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:26.258725 11261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:26.258864 11261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:26.266013 11261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:26.266155 11261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:26.266242 11261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:26.266315 11261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:26.266378 11261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:26.276389 11261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:44661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:40093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:26.277755 11261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:26.279202 11261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:26.291275 11266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:26.291193 11269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:26.291251 11267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:26.292227 11261 server_base.cc:1034] running on GCE node
I20260923 17:28:26.293031 11261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:26.294615 11261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:26.295836 11261 hybrid_clock.cc:648] HybridClock initialized: now 1790184506295775 us; error 55 us; skew 500 ppm
I20260923 17:28:26.296257 11261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:26 dist-test-slave-p7jt krb5kdc[11248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184506, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:26.305245 11261 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:26.306897 11261 webserver.cc:460] Webserver started at http://127.8.29.254:46349/ using document root <none> and password file <none>
I20260923 17:28:26.307605 11261 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:26.307726 11261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:26.308028 11261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:26.311201 11261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "2535ee6b6d11418ba61c1d71b28013e0"
format_stamp: "Formatted at 2026-09-23 17:28:26 on dist-test-slave-p7jt"
server_key: "94b4d031fb4d4a21582b42748e187c34"
server_key_iv: "2018c72bb0725f7e8c356414f2b0df76"
server_key_version: "encryptionkey@0"
I20260923 17:28:26.312005 11261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "2535ee6b6d11418ba61c1d71b28013e0"
format_stamp: "Formatted at 2026-09-23 17:28:26 on dist-test-slave-p7jt"
server_key: "94b4d031fb4d4a21582b42748e187c34"
server_key_iv: "2018c72bb0725f7e8c356414f2b0df76"
server_key_version: "encryptionkey@0"
I20260923 17:28:26.317420 11261 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:28:26.321102 11276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:26.322713 11261 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:28:26.322878 11261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "2535ee6b6d11418ba61c1d71b28013e0"
format_stamp: "Formatted at 2026-09-23 17:28:26 on dist-test-slave-p7jt"
server_key: "94b4d031fb4d4a21582b42748e187c34"
server_key_iv: "2018c72bb0725f7e8c356414f2b0df76"
server_key_version: "encryptionkey@0"
I20260923 17:28:26.323071 11261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:26.364966 11261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:26.369354 11261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:26.369629 11261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:26.386864 11261 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:40093
I20260923 17:28:26.386901 11327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:40093 every 8 connection(s)
I20260923 17:28:26.388226 11261 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:26.390338 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 11261
I20260923 17:28:26.390609 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:26.391353 8311 external_mini_cluster.cc:1428] Setting key be9efa1bd167600b7201685ea432561e
I20260923 17:28:26.392901 11328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:28:26 dist-test-slave-p7jt krb5kdc[11248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184505, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:26.406767 11328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0: Bootstrap starting.
I20260923 17:28:26.410080 11328 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:26.411347 11328 log.cc:824] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:26.414718 11328 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0: No bootstrap required, opened a new log
I20260923 17:28:26.414794 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:26.393300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.8.29.254:40093)
0923 17:28:26.394051 (+ 751us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:26.394063 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:26.394103 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:28:26.395715 (+ 1612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:26.395738 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:26.396158 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:26.396512 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:26.398328 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:26.399773 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:26.401467 (+ 1694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:26.402234 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:26.402414 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:26.407175 (+ 4761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:26.407184 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:26.407201 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:26.407245 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:26.410188 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:26.411053 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:26.411060 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:26.411071 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:26.411250 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:26.411703 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:26.411708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:26.411710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:26.412605 (+ 895us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:26.412896 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:26.413295 (+ 399us) server_negotiation.cc:299] Negotiation successful
0923 17:28:26.413689 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":336,"threads_started":1}
I20260923 17:28:26.419050 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:26.422717 11328 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2535ee6b6d11418ba61c1d71b28013e0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40093 } }
I20260923 17:28:26.423143 11328 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:26.423238 11328 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2535ee6b6d11418ba61c1d71b28013e0, State: Initialized, Role: FOLLOWER
I20260923 17:28:26.423811 11328 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2535ee6b6d11418ba61c1d71b28013e0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40093 } }
I20260923 17:28:26.424037 11328 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:26.424155 11328 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:26.424319 11328 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:26.426905 11328 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2535ee6b6d11418ba61c1d71b28013e0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40093 } }
I20260923 17:28:26.427457 11328 leader_election.cc:302] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2535ee6b6d11418ba61c1d71b28013e0; no voters:
I20260923 17:28:26.428020 11328 leader_election.cc:288] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:26.428174 11334 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:26.430007 11328 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:26.430719 11334 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [term 1 LEADER]: Becoming Leader. State: Replica: 2535ee6b6d11418ba61c1d71b28013e0, State: Running, Role: LEADER
I20260923 17:28:26.431442 11334 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2535ee6b6d11418ba61c1d71b28013e0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40093 } }
I20260923 17:28:26.435439 11336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2535ee6b6d11418ba61c1d71b28013e0. Latest consensus state: current_term: 1 leader_uuid: "2535ee6b6d11418ba61c1d71b28013e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2535ee6b6d11418ba61c1d71b28013e0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40093 } } }
I20260923 17:28:26.435745 11336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:26.437395 11340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:26.438731 11335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2535ee6b6d11418ba61c1d71b28013e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2535ee6b6d11418ba61c1d71b28013e0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40093 } } }
I20260923 17:28:26.439042 11335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:26.444554 11340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:26.455128 11340 catalog_manager.cc:1347] Generated new cluster ID: b6f0aaaef15249e6b0ec16e11abd7bcd
I20260923 17:28:26.455255 11340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:26.469376 11340 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:26.471369 11340 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:26.485723 11340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2535ee6b6d11418ba61c1d71b28013e0: Generated new TSK 0
I20260923 17:28:26.487104 11340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:26.505126 11331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:26.496527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59766 (local address 127.8.29.254:40093)
0923 17:28:26.496790 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:26.496798 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:26.497913 (+ 1115us) server_negotiation.cc:407] Connection header received
0923 17:28:26.498107 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:26.498111 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:26.498184 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:26.498275 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:26.499344 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:26.500124 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:26.501492 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:26.501939 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:26.502063 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:26.503508 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:26.503514 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:26.503517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:26.504245 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:26.504300 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:26.504335 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:26.504448 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:26.504640 (+ 192us) server_negotiation.cc:299] Negotiation successful
0923 17:28:26.504753 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:28:26.513355 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11261
I20260923 17:28:26.514621 11323 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:26.611048 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 11261
2026-09-23T17:28:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1862 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:26 dist-test-slave-p7jt krb5kdc[11362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:26 dist-test-slave-p7jt krb5kdc[11362](info): set up 2 sockets
Sep 23 17:28:26 dist-test-slave-p7jt krb5kdc[11362](info): commencing operation
krb5kdc: starting...
W20260923 17:28:27.602535 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.958s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:27 dist-test-slave-p7jt krb5kdc[11362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:28:27.790040 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:33929
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:33929
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:28.105605 11375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:28.105892 11375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:28.105978 11375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:28.113415 11375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:28.113543 11375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:28.113587 11375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:28.113643 11375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:28.113685 11375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:28.123787 11375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:33929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:33929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:28.125090 11375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:28.126345 11375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:28.137986 11380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:28.138015 11383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:28.138036 11381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:28.138945 11375 server_base.cc:1034] running on GCE node
I20260923 17:28:28.139799 11375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:28.141312 11375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:28.142535 11375 hybrid_clock.cc:648] HybridClock initialized: now 1790184508142484 us; error 43 us; skew 500 ppm
I20260923 17:28:28.142899 11375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:28 dist-test-slave-p7jt krb5kdc[11362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184508, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:28.145983 11375 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:28.147509 11375 webserver.cc:460] Webserver started at http://127.8.29.254:38235/ using document root <none> and password file <none>
I20260923 17:28:28.148166 11375 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:28.148265 11375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:28.148518 11375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:28.151495 11375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "39194448e99248fcbaf80ecb4b286b17"
format_stamp: "Formatted at 2026-09-23 17:28:28 on dist-test-slave-p7jt"
server_key: "6d1454edbd7fad9a5f1118ba7dee902f"
server_key_iv: "5f316f118c5f4cd04f3aa6e9ac973a6d"
server_key_version: "encryptionkey@0"
I20260923 17:28:28.152200 11375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "39194448e99248fcbaf80ecb4b286b17"
format_stamp: "Formatted at 2026-09-23 17:28:28 on dist-test-slave-p7jt"
server_key: "6d1454edbd7fad9a5f1118ba7dee902f"
server_key_iv: "5f316f118c5f4cd04f3aa6e9ac973a6d"
server_key_version: "encryptionkey@0"
I20260923 17:28:28.157497 11375 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 17:28:28.160907 11390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:28.162256 11375 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:28.162412 11375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "39194448e99248fcbaf80ecb4b286b17"
format_stamp: "Formatted at 2026-09-23 17:28:28 on dist-test-slave-p7jt"
server_key: "6d1454edbd7fad9a5f1118ba7dee902f"
server_key_iv: "5f316f118c5f4cd04f3aa6e9ac973a6d"
server_key_version: "encryptionkey@0"
I20260923 17:28:28.162647 11375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:28.182920 11375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:28.187243 11375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:28.187517 11375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:28.203791 11375 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:33929
I20260923 17:28:28.203814 11441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:33929 every 8 connection(s)
I20260923 17:28:28.205168 11375 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:28.207363 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 11375
I20260923 17:28:28.207556 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:28.208186 8311 external_mini_cluster.cc:1428] Setting key 473e7ec7975587b0753b329057c4ba05
I20260923 17:28:28.209465 11442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:28:28 dist-test-slave-p7jt krb5kdc[11362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184507, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:28.223227 11442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17: Bootstrap starting.
I20260923 17:28:28.226531 11442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:28.227548 11442 log.cc:824] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:28.230423 11442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17: No bootstrap required, opened a new log
I20260923 17:28:28.230674 11445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:28.210195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51342 (local address 127.8.29.254:33929)
0923 17:28:28.210981 (+ 786us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:28.210998 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:28.211051 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:28:28.212893 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:28.212922 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:28.213457 (+ 535us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:28.213932 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:28.215202 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:28.217213 (+ 2011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:28.218542 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:28.219566 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:28.219750 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:28.223610 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:28.223619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:28.223636 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:28.223674 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:28.226644 (+ 2970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:28.227335 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:28.227342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:28.227349 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:28.227515 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:28.227921 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:28.227925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:28.227927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:28.228470 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:28.228769 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:28.229209 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 17:28:28.229621 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":343,"threads_started":1}
I20260923 17:28:28.238804 11442 raft_consensus.cc:348] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39194448e99248fcbaf80ecb4b286b17" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33929 } }
I20260923 17:28:28.239329 11442 raft_consensus.cc:374] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:28.239432 11442 raft_consensus.cc:729] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39194448e99248fcbaf80ecb4b286b17, State: Initialized, Role: FOLLOWER
I20260923 17:28:28.239993 11442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39194448e99248fcbaf80ecb4b286b17" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33929 } }
I20260923 17:28:28.240221 11442 raft_consensus.cc:388] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:28.240396 11442 raft_consensus.cc:482] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:28.240558 11442 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:28.242733 11442 raft_consensus.cc:504] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39194448e99248fcbaf80ecb4b286b17" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33929 } }
I20260923 17:28:28.243232 11442 leader_election.cc:302] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39194448e99248fcbaf80ecb4b286b17; no voters:
I20260923 17:28:28.243805 11442 leader_election.cc:288] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:28.243959 11447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:28.244589 11447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [term 1 LEADER]: Becoming Leader. State: Replica: 39194448e99248fcbaf80ecb4b286b17, State: Running, Role: LEADER
I20260923 17:28:28.245162 11447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39194448e99248fcbaf80ecb4b286b17" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33929 } }
I20260923 17:28:28.245841 11442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:28.247705 11449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39194448e99248fcbaf80ecb4b286b17. Latest consensus state: current_term: 1 leader_uuid: "39194448e99248fcbaf80ecb4b286b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39194448e99248fcbaf80ecb4b286b17" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33929 } } }
I20260923 17:28:28.247995 11449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:28.248149 11448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39194448e99248fcbaf80ecb4b286b17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39194448e99248fcbaf80ecb4b286b17" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33929 } } }
I20260923 17:28:28.248523 11448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:28.250414 11452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:28.255997 11452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:28.266386 11452 catalog_manager.cc:1347] Generated new cluster ID: 3b77a763767147e1bebb3faea20bd3b3
I20260923 17:28:28.266539 11452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:28.289398 11452 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:28.290822 11452 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:28.300127 11452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 39194448e99248fcbaf80ecb4b286b17: Generated new TSK 0
I20260923 17:28:28.300984 11452 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:28:28.353965 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33929
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:28.676376 11468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:28.676669 11468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:28.676764 11468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:28.685141 11468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:28.685256 11468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:28.685374 11468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:28.695282 11468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:28.696545 11468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:28.697774 11468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:28.710970 11476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:28.711032 11474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:28.711023 11473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:28.711751 11468 server_base.cc:1034] running on GCE node
I20260923 17:28:28.712364 11468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:28.713347 11468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:28.714547 11468 hybrid_clock.cc:648] HybridClock initialized: now 1790184508714474 us; error 58 us; skew 500 ppm
I20260923 17:28:28.714857 11468 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:28 dist-test-slave-p7jt krb5kdc[11362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184508, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:28.717881 11468 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:28:28.719404 11468 webserver.cc:460] Webserver started at http://127.8.29.193:37489/ using document root <none> and password file <none>
I20260923 17:28:28.720086 11468 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:28.720198 11468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:28.720486 11468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:28.723457 11468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "33ec7e1223d14ccea8da610dd26373e8"
format_stamp: "Formatted at 2026-09-23 17:28:28 on dist-test-slave-p7jt"
server_key: "2c14374f15fdf633d04623919629e338"
server_key_iv: "91c229c4fe9b8a117e97e4ae5cd77ebe"
server_key_version: "encryptionkey@0"
I20260923 17:28:28.724211 11468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "33ec7e1223d14ccea8da610dd26373e8"
format_stamp: "Formatted at 2026-09-23 17:28:28 on dist-test-slave-p7jt"
server_key: "2c14374f15fdf633d04623919629e338"
server_key_iv: "91c229c4fe9b8a117e97e4ae5cd77ebe"
server_key_version: "encryptionkey@0"
I20260923 17:28:28.729379 11468 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:28.732973 11483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:28.734395 11468 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:28:28.734579 11468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "33ec7e1223d14ccea8da610dd26373e8"
format_stamp: "Formatted at 2026-09-23 17:28:28 on dist-test-slave-p7jt"
server_key: "2c14374f15fdf633d04623919629e338"
server_key_iv: "91c229c4fe9b8a117e97e4ae5cd77ebe"
server_key_version: "encryptionkey@0"
I20260923 17:28:28.734778 11468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:28.759569 11468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:28.763432 11468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:28.763715 11468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:28.764670 11468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:28.766016 11468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:28.766100 11468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:28.766217 11468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:28.766275 11468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:28.795878 11468 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:38449
I20260923 17:28:28.795965 11595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:38449 every 8 connection(s)
I20260923 17:28:28.797116 11468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:28.801608 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 11468
I20260923 17:28:28.801803 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:28.802412 8311 external_mini_cluster.cc:1428] Setting key 063e1d653fd7dc19fa6c09bbbc03c912
Sep 23 17:28:28 dist-test-slave-p7jt krb5kdc[11362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184508, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:28.817271 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:28.799522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:35941 (local address 127.8.29.254:33929)
0923 17:28:28.799980 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:28.799985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:28.800944 (+ 959us) server_negotiation.cc:407] Connection header received
0923 17:28:28.802929 (+ 1985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:28.802935 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:28.803013 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:28.803118 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:28.804648 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:28.805632 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:28.807185 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:28.807728 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:28.807840 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:28.811639 (+ 3799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:28.811645 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:28.811649 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:28.811682 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:28.813995 (+ 2313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:28.814916 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:28.814920 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:28.814922 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:28.815078 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:28.815517 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:28.815520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:28.815522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:28.815823 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:28.815964 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:28.816836 (+ 872us) server_negotiation.cc:299] Negotiation successful
0923 17:28:28.817006 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":265,"threads_started":1}
I20260923 17:28:28.818812 11599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:28.800109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33929 (local address 127.8.29.193:35941)
0923 17:28:28.800796 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:28.800835 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:28.802674 (+ 1839us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:28.803335 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:28.803343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:28.803841 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:28.804484 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:28.804501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:28.805762 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:28.805765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:28.807045 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:28.807057 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:28.807900 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:28.807904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:28.808095 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:28.808719 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:28.808741 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:28.811411 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:28.814144 (+ 2733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:28.814153 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:28.814170 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:28.814760 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:28.815190 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:28.815195 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:28.815198 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:28.815426 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:28.815966 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:28.815976 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:28.816429 (+ 453us) client_negotiation.cc:769] Sending connection context
0923 17:28:28.816762 (+ 333us) client_negotiation.cc:240] Negotiation successful
0923 17:28:28.817333 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":251,"threads_started":1}
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:28:28.820303 11596 heartbeater.cc:347] Connected to a master server at 127.8.29.254:33929
I20260923 17:28:28.820590 11596 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:28.821359 11596 heartbeater.cc:511] Master 127.8.29.254:33929 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
I20260923 17:28:28.823551 11407 ts_manager.cc:194] Registered new tserver with Master: 33ec7e1223d14ccea8da610dd26373e8 (127.8.29.193:38449)
I20260923 17:28:28.825428 11407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:35941
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:28:28.846895 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33929
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:29.159396 11602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:29.159672 11602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:29.159791 11602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:29.166898 11602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:29.167001 11602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:29.167145 11602 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:29.176934 11602 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:29.178225 11602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:29.179534 11602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:29.192031 11607 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:29.192150 11608 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:29.192112 11610 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:29.193382 11602 server_base.cc:1034] running on GCE node
I20260923 17:28:29.193975 11602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:29.194942 11602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:29.196148 11602 hybrid_clock.cc:648] HybridClock initialized: now 1790184509196095 us; error 40 us; skew 500 ppm
I20260923 17:28:29.196460 11602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:29 dist-test-slave-p7jt krb5kdc[11362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184509, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:29.199528 11602 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:28:29.200963 11602 webserver.cc:460] Webserver started at http://127.8.29.194:45215/ using document root <none> and password file <none>
I20260923 17:28:29.201627 11602 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:29.201733 11602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:29.202019 11602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:29.204902 11602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "acd269e2976d4305a181a24575bd348c"
format_stamp: "Formatted at 2026-09-23 17:28:29 on dist-test-slave-p7jt"
server_key: "e75bfd98193421456f966e4fc742c073"
server_key_iv: "2ae90c78926382d15f926a0ffca746db"
server_key_version: "encryptionkey@0"
I20260923 17:28:29.205657 11602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "acd269e2976d4305a181a24575bd348c"
format_stamp: "Formatted at 2026-09-23 17:28:29 on dist-test-slave-p7jt"
server_key: "e75bfd98193421456f966e4fc742c073"
server_key_iv: "2ae90c78926382d15f926a0ffca746db"
server_key_version: "encryptionkey@0"
I20260923 17:28:29.211004 11602 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:28:29.214772 11617 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:29.216442 11602 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:29.216616 11602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "acd269e2976d4305a181a24575bd348c"
format_stamp: "Formatted at 2026-09-23 17:28:29 on dist-test-slave-p7jt"
server_key: "e75bfd98193421456f966e4fc742c073"
server_key_iv: "2ae90c78926382d15f926a0ffca746db"
server_key_version: "encryptionkey@0"
I20260923 17:28:29.216804 11602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:29.230863 11602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:29.235605 11602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:29.235893 11602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:29.236903 11602 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:29.238373 11602 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:29.238451 11602 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:29.238601 11602 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:29.238656 11602 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:29.270923 11602 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:42159
I20260923 17:28:29.270974 11729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:42159 every 8 connection(s)
I20260923 17:28:29.272224 11602 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:29.274139 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 11602
I20260923 17:28:29.274351 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:29.274816 8311 external_mini_cluster.cc:1428] Setting key cd71d7b2331e0b6f45bc4465ed68ea59
Sep 23 17:28:29 dist-test-slave-p7jt krb5kdc[11362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184509, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
I20260923 17:28:29.294767 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.275073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:50201 (local address 127.8.29.254:33929)
0923 17:28:29.275281 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:29.275287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:29.276434 (+ 1147us) server_negotiation.cc:407] Connection header received
0923 17:28:29.278861 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:29.278866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:29.278935 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:29.279025 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:29.281040 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.281802 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.283713 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.284327 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.284457 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:29.288038 (+ 3581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:29.288044 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:29.288049 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:29.288090 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:29.291369 (+ 3279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.292265 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.292271 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.292275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.292435 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.292939 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.292944 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.292948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.293305 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:29.293460 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:29.294266 (+ 806us) server_negotiation.cc:299] Negotiation successful
0923 17:28:29.294446 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:28:29.295776 11732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.275568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33929 (local address 127.8.29.194:50201)
0923 17:28:29.276283 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:29.276329 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:29.278524 (+ 2195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:29.279314 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:29.279326 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:29.280017 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:29.280836 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:29.280864 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.281954 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.281959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:29.283552 (+ 1593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:29.283565 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.284525 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.284529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:29.284735 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:29.285386 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:29.285409 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:29.287780 (+ 2371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:29.291560 (+ 3780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:29.291568 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:29.291584 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:29.292116 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:29.292578 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:29.292582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:29.292584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:29.292806 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:29.293460 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:29.293467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:29.293862 (+ 395us) client_negotiation.cc:769] Sending connection context
0923 17:28:29.294163 (+ 301us) client_negotiation.cc:240] Negotiation successful
0923 17:28:29.294650 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":289,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
I20260923 17:28:29.297386 11730 heartbeater.cc:347] Connected to a master server at 127.8.29.254:33929
I20260923 17:28:29.297712 11730 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:29.298429 11730 heartbeater.cc:511] Master 127.8.29.254:33929 requested a full tablet report, sending...
I20260923 17:28:29.299950 11407 ts_manager.cc:194] Registered new tserver with Master: acd269e2976d4305a181a24575bd348c (127.8.29.194:42159)
I20260923 17:28:29.300779 11407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:50201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:28:29.321732 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33929
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:29.640033 11735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:29.640341 11735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:29.640478 11735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:29.647734 11735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:29.647861 11735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:29.648015 11735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:29.657804 11735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33929
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:29.659147 11735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:29.660431 11735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:29.672799 11740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:29.672827 11743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:29.672966 11741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:29.673476 11735 server_base.cc:1034] running on GCE node
I20260923 17:28:29.674094 11735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:29.675056 11735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:29.676275 11735 hybrid_clock.cc:648] HybridClock initialized: now 1790184509676212 us; error 52 us; skew 500 ppm
I20260923 17:28:29.676575 11735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:29 dist-test-slave-p7jt krb5kdc[11362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184509, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:29.679648 11735 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:28:29.681085 11735 webserver.cc:460] Webserver started at http://127.8.29.195:41781/ using document root <none> and password file <none>
I20260923 17:28:29.681735 11735 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:29.681836 11735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:29.682101 11735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:29.685822 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "0b02c82192c6498cb9271f833ae89829"
format_stamp: "Formatted at 2026-09-23 17:28:29 on dist-test-slave-p7jt"
server_key: "dc67aa52b836612491f3bd8877e61c3d"
server_key_iv: "54c478745865bb81b61b361fd5c4bd19"
server_key_version: "encryptionkey@0"
I20260923 17:28:29.686743 11735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "0b02c82192c6498cb9271f833ae89829"
format_stamp: "Formatted at 2026-09-23 17:28:29 on dist-test-slave-p7jt"
server_key: "dc67aa52b836612491f3bd8877e61c3d"
server_key_iv: "54c478745865bb81b61b361fd5c4bd19"
server_key_version: "encryptionkey@0"
I20260923 17:28:29.691864 11735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260923 17:28:29.695466 11750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:29.696841 11735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:28:29.696985 11735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "0b02c82192c6498cb9271f833ae89829"
format_stamp: "Formatted at 2026-09-23 17:28:29 on dist-test-slave-p7jt"
server_key: "dc67aa52b836612491f3bd8877e61c3d"
server_key_iv: "54c478745865bb81b61b361fd5c4bd19"
server_key_version: "encryptionkey@0"
I20260923 17:28:29.697183 11735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:29.732499 11735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:29.737438 11735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:29.737732 11735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:29.738847 11735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:29.740367 11735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:29.740447 11735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:29.740573 11735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:29.740621 11735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:29.771389 11735 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:43037
I20260923 17:28:29.771437 11862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:43037 every 8 connection(s)
I20260923 17:28:29.772689 11735 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:29.779577 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 11735
I20260923 17:28:29.779784 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:29.780372 8311 external_mini_cluster.cc:1428] Setting key f64d8078921c4b0ebbd997a25dcc3617
Sep 23 17:28:29 dist-test-slave-p7jt krb5kdc[11362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184509, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:29.793396 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.775282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:56145 (local address 127.8.29.254:33929)
0923 17:28:29.775454 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:29.775459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:29.776645 (+ 1186us) server_negotiation.cc:407] Connection header received
0923 17:28:29.778536 (+ 1891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:29.778543 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:29.778628 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:29.778744 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:29.780973 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.781982 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.783748 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.784169 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.784271 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:29.787775 (+ 3504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:29.787780 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:29.787783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:29.787816 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:29.790293 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.791148 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.791153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.791155 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.791276 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.791742 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.791745 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.791748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.792070 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:29.792183 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:29.792954 (+ 771us) server_negotiation.cc:299] Negotiation successful
0923 17:28:29.793144 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:28:29.794520 11865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.775771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33929 (local address 127.8.29.195:56145)
0923 17:28:29.776469 (+ 698us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:29.776514 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:29.778175 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:29.779071 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:29.779085 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:29.779847 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:29.780720 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:29.780751 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.782161 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.782167 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:29.783583 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:29.783599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.784363 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.784368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:29.784576 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:29.785168 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:29.785195 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:29.787568 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:29.790506 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:29.790516 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:29.790536 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:29.791031 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:29.791398 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:29.791419 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:29.791421 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:29.791631 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:29.792186 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:29.792193 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:29.792561 (+ 368us) client_negotiation.cc:769] Sending connection context
0923 17:28:29.792809 (+ 248us) client_negotiation.cc:240] Negotiation successful
0923 17:28:29.793227 (+ 418us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":278,"threads_started":1}
I20260923 17:28:29.795991 11863 heartbeater.cc:347] Connected to a master server at 127.8.29.254:33929
I20260923 17:28:29.796335 11863 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:29.797143 11863 heartbeater.cc:511] Master 127.8.29.254:33929 requested a full tablet report, sending...
I20260923 17:28:29.798655 11407 ts_manager.cc:194] Registered new tserver with Master: 0b02c82192c6498cb9271f833ae89829 (127.8.29.195:43037)
I20260923 17:28:29.799754 11407 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:56145
I20260923 17:28:29.808835 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:29.828850 11596 heartbeater.cc:503] Master 127.8.29.254:33929 was elected leader, sending a full tablet report...
Sep 23 17:28:29 dist-test-slave-p7jt krb5kdc[11362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184509, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 17:28:29 dist-test-slave-p7jt krb5kdc[11362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184509, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:29.855274 11598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.841091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.8.29.254:33929)
0923 17:28:29.841297 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:29.841305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:29.841700 (+ 395us) server_negotiation.cc:407] Connection header received
0923 17:28:29.841963 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:29.841968 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:29.842049 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:29.842150 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:29.843506 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.844318 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.845557 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.845972 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.846097 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:29.849846 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:29.849851 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:29.849855 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:29.849893 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:29.852410 (+ 2517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.853204 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.853209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.853211 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.853367 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.853802 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.853806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.853808 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.854180 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:29.854369 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:29.854769 (+ 400us) server_negotiation.cc:299] Negotiation successful
0923 17:28:29.854949 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260923 17:28:29.855886 11407 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:51358
I20260923 17:28:29.858008 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11468
Sep 23 17:28:29 dist-test-slave-p7jt krb5kdc[11362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184509, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:29.873607 11874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.859010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.8.29.193:38449)
0923 17:28:29.859720 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:29.859727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:29.859751 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:28:29.859813 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:29.859820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:29.860030 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:29.860193 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:29.861363 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.862377 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.863610 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.864179 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.864295 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:29.868163 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:29.868175 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:29.868190 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:29.868224 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:29.870907 (+ 2683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.871568 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.871576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.871578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.871705 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.872058 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.872063 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.872065 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.872403 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:29.872729 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:29.873004 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 17:28:29.873265 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":334,"threads_started":1}
W20260923 17:28:29.874907 11591 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:34656
W20260923 17:28:29.876122 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11468: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:28:29.876310 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 11468
I20260923 17:28:29.888828 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11602
Sep 23 17:28:29 dist-test-slave-p7jt krb5kdc[11362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184509, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:28:29.904541 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.889758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.8.29.194:42159)
0923 17:28:29.890342 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:29.890349 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:29.890376 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:28:29.890438 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:29.890446 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:29.890735 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:29.890864 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:29.892115 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.893192 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.894489 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.895084 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.895184 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:29.899228 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:29.899241 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:29.899259 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:29.899297 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:29.902082 (+ 2785us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.902730 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.902737 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.902739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.902856 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.903151 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.903154 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.903156 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.903492 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:29.903724 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:29.904000 (+ 276us) server_negotiation.cc:299] Negotiation successful
0923 17:28:29.904202 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":340,"threads_started":1}
W20260923 17:28:29.905849 11725 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36456
W20260923 17:28:29.906987 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11602: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:28:29.907166 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 11602
I20260923 17:28:29.919378 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11735
Sep 23 17:28:29 dist-test-slave-p7jt krb5kdc[11362](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184509, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:28:29.935986 11876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:29.920397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35920 (local address 127.8.29.195:43037)
0923 17:28:29.921000 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:29.921008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:29.921037 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:28:29.921097 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:29.921104 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:29.921308 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:29.921431 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:29.922828 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.923923 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.925110 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:29.925671 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:29.925766 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:29.929939 (+ 4173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:29.929953 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:29.929968 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:29.930012 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:29.932681 (+ 2669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.933788 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.933797 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.933799 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.933942 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:29.934361 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:29.934365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:29.934367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:29.934854 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:29.935119 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:29.935444 (+ 325us) server_negotiation.cc:299] Negotiation successful
0923 17:28:29.935698 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":303,"threads_started":1}
W20260923 17:28:29.937343 11858 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35920
W20260923 17:28:29.938643 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11735: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:28:29.938822 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 11735
I20260923 17:28:29.951484 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11375
I20260923 17:28:29.952853 11437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:30.055352 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 11375
2026-09-23T17:28:30Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3448 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:30 dist-test-slave-p7jt krb5kdc[11880](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:30 dist-test-slave-p7jt krb5kdc[11880](info): set up 2 sockets
Sep 23 17:28:30 dist-test-slave-p7jt krb5kdc[11880](info): commencing operation
krb5kdc: starting...
W20260923 17:28:31.062767 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.970s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:31 dist-test-slave-p7jt krb5kdc[11880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:28:31.254928 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:37607
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:37607
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:31.568608 11893 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:31.568933 11893 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:31.569079 11893 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:31.576419 11893 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:31.576562 11893 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:31.576638 11893 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:31.576710 11893 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:31.576771 11893 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:31.586750 11893 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:37607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:37607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:31.588112 11893 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:31.589422 11893 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:31.601158 11899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:31.601270 11898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:31.601135 11901 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:31.602016 11893 server_base.cc:1034] running on GCE node
I20260923 17:28:31.602808 11893 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:31.604216 11893 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:31.605423 11893 hybrid_clock.cc:648] HybridClock initialized: now 1790184511605368 us; error 42 us; skew 500 ppm
I20260923 17:28:31.605764 11893 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:31 dist-test-slave-p7jt krb5kdc[11880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184511, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:31.612097 11893 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:31.613746 11893 webserver.cc:460] Webserver started at http://127.8.29.254:33761/ using document root <none> and password file <none>
I20260923 17:28:31.614519 11893 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:31.614660 11893 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:31.614979 11893 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:31.618086 11893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "47dcbd6aea334392a91ada4c189718f0"
format_stamp: "Formatted at 2026-09-23 17:28:31 on dist-test-slave-p7jt"
server_key: "50d5638fd484b21e52a874f722db9fbb"
server_key_iv: "926ce9ea1c1c2f324298b1c65afa9713"
server_key_version: "encryptionkey@0"
I20260923 17:28:31.618914 11893 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "47dcbd6aea334392a91ada4c189718f0"
format_stamp: "Formatted at 2026-09-23 17:28:31 on dist-test-slave-p7jt"
server_key: "50d5638fd484b21e52a874f722db9fbb"
server_key_iv: "926ce9ea1c1c2f324298b1c65afa9713"
server_key_version: "encryptionkey@0"
I20260923 17:28:31.624267 11893 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:28:31.627861 11908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:31.629305 11893 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:31.629485 11893 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "47dcbd6aea334392a91ada4c189718f0"
format_stamp: "Formatted at 2026-09-23 17:28:31 on dist-test-slave-p7jt"
server_key: "50d5638fd484b21e52a874f722db9fbb"
server_key_iv: "926ce9ea1c1c2f324298b1c65afa9713"
server_key_version: "encryptionkey@0"
I20260923 17:28:31.629678 11893 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:31.649731 11893 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:31.654304 11893 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:31.654619 11893 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:31.671888 11893 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:37607
I20260923 17:28:31.671911 11959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:37607 every 8 connection(s)
I20260923 17:28:31.673288 11893 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:31.677166 11960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:31.683068 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 11893
I20260923 17:28:31.683288 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:31.684092 8311 external_mini_cluster.cc:1428] Setting key 7aff49a5feae983478825edd08f1b591
I20260923 17:28:31.689219 11960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0: Bootstrap starting.
I20260923 17:28:31.694059 11960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:31.695662 11960 log.cc:824] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:31.699539 11960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0: No bootstrap required, opened a new log
Sep 23 17:28:31 dist-test-slave-p7jt krb5kdc[11880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184511, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:31.709479 11960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dcbd6aea334392a91ada4c189718f0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37607 } }
I20260923 17:28:31.709877 11960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:31.709986 11960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47dcbd6aea334392a91ada4c189718f0, State: Initialized, Role: FOLLOWER
I20260923 17:28:31.710686 11960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dcbd6aea334392a91ada4c189718f0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37607 } }
I20260923 17:28:31.710924 11960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:31.711078 11960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:31.711301 11960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:31.713388 11963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:31.686268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57474 (local address 127.8.29.254:37607)
0923 17:28:31.687080 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:31.687093 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:31.687134 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:28:31.689381 (+ 2247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:31.689420 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:31.690011 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:31.690511 (+ 500us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:31.692447 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:31.694415 (+ 1968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:31.696306 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:31.697365 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:31.697550 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:31.702659 (+ 5109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:31.702672 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:31.702699 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:31.702763 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:31.707268 (+ 4505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:31.708211 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:31.708223 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:31.708235 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:31.708503 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:31.708959 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:31.708965 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:31.708968 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:31.709822 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:31.710108 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:31.711477 (+ 1369us) server_negotiation.cc:299] Negotiation successful
0923 17:28:31.711992 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":393,"threads_started":1}
I20260923 17:28:31.714586 11960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dcbd6aea334392a91ada4c189718f0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37607 } }
I20260923 17:28:31.715234 11960 leader_election.cc:302] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47dcbd6aea334392a91ada4c189718f0; no voters:
I20260923 17:28:31.715936 11960 leader_election.cc:288] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:31.716269 11965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:31.717227 11965 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [term 1 LEADER]: Becoming Leader. State: Replica: 47dcbd6aea334392a91ada4c189718f0, State: Running, Role: LEADER
I20260923 17:28:31.718155 11960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:31.718161 11965 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dcbd6aea334392a91ada4c189718f0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37607 } }
I20260923 17:28:31.721107 11967 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47dcbd6aea334392a91ada4c189718f0. Latest consensus state: current_term: 1 leader_uuid: "47dcbd6aea334392a91ada4c189718f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dcbd6aea334392a91ada4c189718f0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37607 } } }
I20260923 17:28:31.721129 11966 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47dcbd6aea334392a91ada4c189718f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47dcbd6aea334392a91ada4c189718f0" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37607 } } }
I20260923 17:28:31.721555 11966 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:31.721555 11967 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:31.725761 11970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:31.731151 11970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:31.746701 11970 catalog_manager.cc:1347] Generated new cluster ID: bd1a5d55970a45039f15f8b6db74b04e
I20260923 17:28:31.746838 11970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:31.765228 11970 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:31.766885 11970 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:31.773633 11970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47dcbd6aea334392a91ada4c189718f0: Generated new TSK 0
I20260923 17:28:31.774605 11970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:28:31.821695 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:37607
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:32.147605 11986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:32.147972 11986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:32.148083 11986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:32.155289 11986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:32.155400 11986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:32.155512 11986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:32.165266 11986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:37607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:32.166580 11986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:32.167898 11986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:32.181804 11992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:32.181780 11994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:32.181751 11991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:32.182538 11986 server_base.cc:1034] running on GCE node
I20260923 17:28:32.183138 11986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:32.184108 11986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:32.185317 11986 hybrid_clock.cc:648] HybridClock initialized: now 1790184512185253 us; error 56 us; skew 500 ppm
I20260923 17:28:32.185645 11986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:32 dist-test-slave-p7jt krb5kdc[11880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184512, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:32.189049 11986 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:28:32.190636 11986 webserver.cc:460] Webserver started at http://127.8.29.193:36453/ using document root <none> and password file <none>
I20260923 17:28:32.191392 11986 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:32.191516 11986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:32.191818 11986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:32.194913 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "86254cb2aa514e5c9b07667d9faa15d0"
format_stamp: "Formatted at 2026-09-23 17:28:32 on dist-test-slave-p7jt"
server_key: "33162445b64d21a6f6276b69c7b974d7"
server_key_iv: "53364b6ba3e6603f5d80d989c52c1b8c"
server_key_version: "encryptionkey@0"
I20260923 17:28:32.195708 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "86254cb2aa514e5c9b07667d9faa15d0"
format_stamp: "Formatted at 2026-09-23 17:28:32 on dist-test-slave-p7jt"
server_key: "33162445b64d21a6f6276b69c7b974d7"
server_key_iv: "53364b6ba3e6603f5d80d989c52c1b8c"
server_key_version: "encryptionkey@0"
I20260923 17:28:32.201267 11986 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:32.205169 12001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:32.206820 11986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:28:32.206995 11986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "86254cb2aa514e5c9b07667d9faa15d0"
format_stamp: "Formatted at 2026-09-23 17:28:32 on dist-test-slave-p7jt"
server_key: "33162445b64d21a6f6276b69c7b974d7"
server_key_iv: "53364b6ba3e6603f5d80d989c52c1b8c"
server_key_version: "encryptionkey@0"
I20260923 17:28:32.207198 11986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:32.240098 11986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:32.244549 11986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:32.244825 11986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:32.245844 11986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:32.247279 11986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:32.247352 11986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:32.247452 11986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:32.247515 11986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:32.277770 11986 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:43575
I20260923 17:28:32.277810 12113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:43575 every 8 connection(s)
I20260923 17:28:32.279143 11986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:32.280457 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 11986
I20260923 17:28:32.280642 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:32.281281 8311 external_mini_cluster.cc:1428] Setting key 193c0e6f9c670b8cdc0d4143ed935efd
Sep 23 17:28:32 dist-test-slave-p7jt krb5kdc[11880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184512, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:28:32.301026 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:32.281836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:59047 (local address 127.8.29.254:37607)
0923 17:28:32.282793 (+ 957us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:32.282799 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:32.283561 (+ 762us) server_negotiation.cc:407] Connection header received
0923 17:28:32.285488 (+ 1927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:32.285493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:32.285581 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:32.285686 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:32.287359 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:32.288189 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:32.289956 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:32.290674 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:32.290823 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:32.294529 (+ 3706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:32.294538 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:32.294544 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:32.294593 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:32.297860 (+ 3267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:32.298609 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:32.298616 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:32.298620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:32.298777 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:32.299245 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:32.299251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:32.299254 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:32.299662 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:32.299809 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:32.300566 (+ 757us) server_negotiation.cc:299] Negotiation successful
0923 17:28:32.300756 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":845,"thread_start_us":737,"threads_started":1}
I20260923 17:28:32.302127 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:32.282591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:37607 (local address 127.8.29.193:59047)
0923 17:28:32.283396 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:32.283441 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:32.285199 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:32.285906 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:32.285915 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:32.286576 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:32.287193 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:32.287213 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:32.288365 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:32.288371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:32.289776 (+ 1405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:32.289790 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:32.290905 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:32.290910 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:32.291129 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:32.291809 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:32.291831 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:32.294208 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:32.298010 (+ 3802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:32.298017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:32.298031 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:32.298440 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:32.298905 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:32.298908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:32.298910 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:32.299120 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:32.299785 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:32.299792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:32.300144 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 17:28:32.300425 (+ 281us) client_negotiation.cc:240] Negotiation successful
0923 17:28:32.300886 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":318,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
I20260923 17:28:32.303788 12114 heartbeater.cc:347] Connected to a master server at 127.8.29.254:37607
I20260923 17:28:32.304149 12114 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:32.305025 12114 heartbeater.cc:511] Master 127.8.29.254:37607 requested a full tablet report, sending...
I20260923 17:28:32.307331 11925 ts_manager.cc:194] Registered new tserver with Master: 86254cb2aa514e5c9b07667d9faa15d0 (127.8.29.193:43575)
I20260923 17:28:32.309250 11925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:59047
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:28:32.326879 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:37607
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:32.652379 12120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:32.652696 12120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:32.652823 12120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:32.660226 12120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:32.660348 12120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:32.660504 12120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:32.670547 12120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:37607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:32.671928 12120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:32.673230 12120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:32.686539 12125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:32.686537 12126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:32.687673 12128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:32.687951 12120 server_base.cc:1034] running on GCE node
I20260923 17:28:32.688612 12120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:32.689620 12120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:32.690935 12120 hybrid_clock.cc:648] HybridClock initialized: now 1790184512690817 us; error 63 us; skew 500 ppm
I20260923 17:28:32.691300 12120 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:32 dist-test-slave-p7jt krb5kdc[11880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184512, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:32.695248 12120 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:28:32.697171 12120 webserver.cc:460] Webserver started at http://127.8.29.194:38495/ using document root <none> and password file <none>
I20260923 17:28:32.697860 12120 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:32.697988 12120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:32.698280 12120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:32.701294 12120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "0a44343ada3444aab5313197511e8617"
format_stamp: "Formatted at 2026-09-23 17:28:32 on dist-test-slave-p7jt"
server_key: "476fcc7cec5c12845f9f481baa3b2533"
server_key_iv: "d5130ec80345d8ca34074e534d09c9bc"
server_key_version: "encryptionkey@0"
I20260923 17:28:32.702091 12120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "0a44343ada3444aab5313197511e8617"
format_stamp: "Formatted at 2026-09-23 17:28:32 on dist-test-slave-p7jt"
server_key: "476fcc7cec5c12845f9f481baa3b2533"
server_key_iv: "d5130ec80345d8ca34074e534d09c9bc"
server_key_version: "encryptionkey@0"
I20260923 17:28:32.707418 12120 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:28:32.710952 12135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:32.712388 12120 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260923 17:28:32.712544 12120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "0a44343ada3444aab5313197511e8617"
format_stamp: "Formatted at 2026-09-23 17:28:32 on dist-test-slave-p7jt"
server_key: "476fcc7cec5c12845f9f481baa3b2533"
server_key_iv: "d5130ec80345d8ca34074e534d09c9bc"
server_key_version: "encryptionkey@0"
I20260923 17:28:32.712751 12120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:32.750257 12120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:32.754593 12120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:32.754861 12120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:32.755853 12120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:32.757210 12120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:32.757285 12120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:32.757395 12120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:32.757445 12120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:32.786834 12120 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:42595
I20260923 17:28:32.786882 12247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:42595 every 8 connection(s)
I20260923 17:28:32.788120 12120 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:32.790184 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 12120
I20260923 17:28:32.790342 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:32.790872 8311 external_mini_cluster.cc:1428] Setting key 6d45e656c67638ae75b5623180110f19
Sep 23 17:28:32 dist-test-slave-p7jt krb5kdc[11880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184512, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
I20260923 17:28:32.808612 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:32.790820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:50123 (local address 127.8.29.254:37607)
0923 17:28:32.791026 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:32.791032 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:32.792442 (+ 1410us) server_negotiation.cc:407] Connection header received
0923 17:28:32.794338 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:32.794344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:32.794431 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:32.794540 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:32.796118 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:32.796833 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:32.798958 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:32.799381 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:32.799493 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:32.803224 (+ 3731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:32.803229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:32.803232 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:32.803267 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:32.805637 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:32.806346 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:32.806350 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:32.806352 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:32.806499 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:32.806904 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:32.806909 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:32.806912 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:32.807203 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:32.807333 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:32.808041 (+ 708us) server_negotiation.cc:299] Negotiation successful
0923 17:28:32.808267 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:28:32.809719 12250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:32.791434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:37607 (local address 127.8.29.194:50123)
0923 17:28:32.792292 (+ 858us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:32.792332 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:32.794052 (+ 1720us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:32.794794 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:32.794803 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:32.795354 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:32.795944 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:32.795963 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:32.796983 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:32.796988 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:32.798812 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:32.798826 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:32.799607 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:32.799616 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:32.799849 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:32.800492 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:32.800516 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:32.802949 (+ 2433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:32.805773 (+ 2824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:32.805783 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:32.805802 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:32.806231 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:32.806610 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:32.806615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:32.806619 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:32.806802 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:32.807324 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:32.807332 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:32.807680 (+ 348us) client_negotiation.cc:769] Sending connection context
0923 17:28:32.807938 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 17:28:32.808492 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":348,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
I20260923 17:28:32.811260 12248 heartbeater.cc:347] Connected to a master server at 127.8.29.254:37607
I20260923 17:28:32.811630 12248 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:32.812531 12248 heartbeater.cc:511] Master 127.8.29.254:37607 requested a full tablet report, sending...
I20260923 17:28:32.814205 11925 ts_manager.cc:194] Registered new tserver with Master: 0a44343ada3444aab5313197511e8617 (127.8.29.194:42595)
I20260923 17:28:32.815294 11925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:50123
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:28:32.834766 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:37607
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:37687
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:33.155450 12253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:33.155750 12253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:33.155874 12253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:33.163442 12253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:33.163578 12253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:33.163724 12253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:33.173859 12253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:37687
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:37607
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:33.175233 12253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:33.176581 12253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:33.189625 12259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:33.189740 12261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:33.189913 12258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:33.190884 12253 server_base.cc:1034] running on GCE node
I20260923 17:28:33.191568 12253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:33.192530 12253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:33.193729 12253 hybrid_clock.cc:648] HybridClock initialized: now 1790184513193671 us; error 59 us; skew 500 ppm
I20260923 17:28:33.194033 12253 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[11880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184513, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:33.197309 12253 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:28:33.198865 12253 webserver.cc:460] Webserver started at http://127.8.29.195:45525/ using document root <none> and password file <none>
I20260923 17:28:33.199563 12253 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:33.199680 12253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:33.199975 12253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:33.203274 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "e12ef1d5f8b345599af54864f48fe4ce"
format_stamp: "Formatted at 2026-09-23 17:28:33 on dist-test-slave-p7jt"
server_key: "cb5347e2c06c6491b7180ac8d26b55c5"
server_key_iv: "1016047b877e3fdf605dd55946eb910e"
server_key_version: "encryptionkey@0"
I20260923 17:28:33.204109 12253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "e12ef1d5f8b345599af54864f48fe4ce"
format_stamp: "Formatted at 2026-09-23 17:28:33 on dist-test-slave-p7jt"
server_key: "cb5347e2c06c6491b7180ac8d26b55c5"
server_key_iv: "1016047b877e3fdf605dd55946eb910e"
server_key_version: "encryptionkey@0"
I20260923 17:28:33.209813 12253 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:28:33.214082 12268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:33.215728 12253 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:33.215931 12253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "e12ef1d5f8b345599af54864f48fe4ce"
format_stamp: "Formatted at 2026-09-23 17:28:33 on dist-test-slave-p7jt"
server_key: "cb5347e2c06c6491b7180ac8d26b55c5"
server_key_iv: "1016047b877e3fdf605dd55946eb910e"
server_key_version: "encryptionkey@0"
I20260923 17:28:33.216233 12253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:33.234710 12253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:33.239454 12253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:33.239759 12253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:33.240768 12253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:33.242194 12253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:33.242271 12253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:33.242396 12253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:33.242447 12253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:33.273438 12253 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:33305
I20260923 17:28:33.273473 12380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:33305 every 8 connection(s)
I20260923 17:28:33.274927 12253 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:33.283331 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 12253
I20260923 17:28:33.283493 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:33.283906 8311 external_mini_cluster.cc:1428] Setting key e1796dc8ea464ebb9d3220e2f8417fef
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[11880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184513, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:33.295336 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.277593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:48235 (local address 127.8.29.254:37607)
0923 17:28:33.277791 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:33.277797 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:33.279092 (+ 1295us) server_negotiation.cc:407] Connection header received
0923 17:28:33.280913 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:33.280918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:33.281038 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:33.281152 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:33.282823 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.283683 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.285560 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.286105 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.286228 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:33.289700 (+ 3472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:33.289705 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:33.289709 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:33.289742 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:33.292216 (+ 2474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.292962 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.292968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.292972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.293096 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.293517 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.293521 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.293523 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.293795 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:33.293930 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:33.294658 (+ 728us) server_negotiation.cc:299] Negotiation successful
0923 17:28:33.294988 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:28:33.296281 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.278075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:37607 (local address 127.8.29.195:48235)
0923 17:28:33.278895 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:33.278960 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:33.280617 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:33.281399 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:33.281410 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:33.282017 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:33.282621 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:33.282640 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.283831 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.283835 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:33.285417 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:33.285431 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.286309 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.286314 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:33.286569 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:33.287306 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:33.287335 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:33.289460 (+ 2125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:33.292356 (+ 2896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:33.292366 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:33.292383 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:33.292836 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:33.293211 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:33.293217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:33.293220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:33.293398 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:33.293929 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:33.293937 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:33.294283 (+ 346us) client_negotiation.cc:769] Sending connection context
0923 17:28:33.294562 (+ 279us) client_negotiation.cc:240] Negotiation successful
0923 17:28:33.295147 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":345,"threads_started":1}
I20260923 17:28:33.297727 12381 heartbeater.cc:347] Connected to a master server at 127.8.29.254:37607
I20260923 17:28:33.298110 12381 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:33.298939 12381 heartbeater.cc:511] Master 127.8.29.254:37607 requested a full tablet report, sending...
I20260923 17:28:33.300311 11925 ts_manager.cc:194] Registered new tserver with Master: e12ef1d5f8b345599af54864f48fe4ce (127.8.29.195:33305)
I20260923 17:28:33.301091 11925 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:48235
I20260923 17:28:33.311849 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:33.312387 12114 heartbeater.cc:503] Master 127.8.29.254:37607 was elected leader, sending a full tablet report...
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[11880](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[11880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:33.361011 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.345280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50714 (local address 127.8.29.193:43575)
0923 17:28:33.346933 (+ 1653us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:33.346940 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:33.346974 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:28:33.347037 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:33.347043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:33.347282 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:33.347417 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:33.348803 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.349841 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.351181 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.351757 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.351857 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:33.355534 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:33.355548 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:33.355565 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:33.355604 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:33.358361 (+ 2757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.359067 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.359083 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.359085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.359218 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.359579 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.359583 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.359585 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.359937 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:33.360205 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:33.360421 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 17:28:33.360627 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1478,"thread_start_us":1330,"threads_started":1}
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[11880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:33.378674 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.365197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.8.29.254:37607)
0923 17:28:33.365389 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:33.365396 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:33.365486 (+ 90us) server_negotiation.cc:407] Connection header received
0923 17:28:33.365660 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:33.365664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:33.365724 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:33.365803 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:33.367101 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.367813 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.369029 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.369420 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.369532 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:33.373630 (+ 4098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:33.373635 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:33.373639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:33.373677 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:33.376042 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.376723 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.376727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.376729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.376861 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.377217 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.377221 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.377223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.377570 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:33.377751 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:33.378184 (+ 433us) server_negotiation.cc:299] Negotiation successful
0923 17:28:33.378353 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 17:28:33.379249 11925 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57490
I20260923 17:28:33.380236 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11986
I20260923 17:28:33.392901 12388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.381046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50720 (local address 127.8.29.193:43575)
0923 17:28:33.381219 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:33.381223 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:33.381483 (+ 260us) server_negotiation.cc:407] Connection header received
0923 17:28:33.381620 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:33.381624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:33.381726 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:33.381805 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:33.383010 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.383716 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.385013 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.385409 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.385501 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:33.387672 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:33.387677 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:33.387681 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:33.387711 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:33.390332 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.391068 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.391072 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.391075 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.391213 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.391647 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.391651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.391653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.391929 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:33.392150 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:33.392457 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 17:28:33.392663 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:28:33.393559 12109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:33.474617 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 11986
I20260923 17:28:33.489410 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 12120
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[11880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:28:33.505584 12393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.490394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46050 (local address 127.8.29.194:42595)
0923 17:28:33.490919 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:33.490926 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:33.490952 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:28:33.491021 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:33.491027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:33.491273 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:33.491414 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:33.492614 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.493640 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.495000 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.495571 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.495672 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:33.499628 (+ 3956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:33.499642 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:33.499658 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:33.499695 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:33.502519 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.503296 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.503304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.503307 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.503458 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.503872 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.503875 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.503878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.504246 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:33.504485 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:33.504857 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 17:28:33.505178 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":280,"threads_started":1}
I20260923 17:28:33.506989 12243 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:33.589257 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 12120
I20260923 17:28:33.601773 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 12253
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[11880](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:28:33.619067 12395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:33.602882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50154 (local address 127.8.29.195:33305)
0923 17:28:33.603552 (+ 670us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:33.603560 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:33.603588 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:28:33.603649 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:33.603656 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:33.603856 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:33.604043 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:33.605305 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.606417 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.607865 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:33.608491 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:33.608631 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:33.613186 (+ 4555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:33.613200 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:33.613216 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:33.613262 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:33.616170 (+ 2908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.616919 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.616927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.616929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.617060 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:33.617411 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:33.617415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:33.617417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:33.617787 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:33.618090 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:33.618401 (+ 311us) server_negotiation.cc:299] Negotiation successful
0923 17:28:33.618713 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":373,"threads_started":1}
I20260923 17:28:33.620441 12376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:33.709353 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 12253
I20260923 17:28:33.721837 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 11893
I20260923 17:28:33.723191 11955 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:33.825488 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 11893
2026-09-23T17:28:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3771 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[12400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[12400](info): set up 2 sockets
Sep 23 17:28:33 dist-test-slave-p7jt krb5kdc[12400](info): commencing operation
krb5kdc: starting...
W20260923 17:28:34.830828 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.967s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:34 dist-test-slave-p7jt krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:28:35.025977 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40381
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:40381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:35.351713 12413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:35.352012 12413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:35.352111 12413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:35.359351 12413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:35.359467 12413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:35.359512 12413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:35.359566 12413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:35.359612 12413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:35.369457 12413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:40381
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:35.370750 12413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:35.372028 12413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:35.383909 12418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:35.384068 12419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:35.384065 12421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:35.385165 12413 server_base.cc:1034] running on GCE node
I20260923 17:28:35.385972 12413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:35.387475 12413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:35.388680 12413 hybrid_clock.cc:648] HybridClock initialized: now 1790184515388631 us; error 39 us; skew 500 ppm
I20260923 17:28:35.389045 12413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:35 dist-test-slave-p7jt krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184515, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:35.392307 12413 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:35.393577 12413 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:35.393676 12413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:35.393993 12413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:35.397185 12413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "604475cc9ce2485f84789280b45f9046"
format_stamp: "Formatted at 2026-09-23 17:28:35 on dist-test-slave-p7jt"
server_key: "311fcdf2ef8efe907148f46422cc5abb"
server_key_iv: "1d5ea28ea2ea7831093d44c43f2eab86"
server_key_version: "encryptionkey@0"
I20260923 17:28:35.398021 12413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "604475cc9ce2485f84789280b45f9046"
format_stamp: "Formatted at 2026-09-23 17:28:35 on dist-test-slave-p7jt"
server_key: "311fcdf2ef8efe907148f46422cc5abb"
server_key_iv: "1d5ea28ea2ea7831093d44c43f2eab86"
server_key_version: "encryptionkey@0"
I20260923 17:28:35.403671 12413 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:28:35.407354 12427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:35.408814 12413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:35.409003 12413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "604475cc9ce2485f84789280b45f9046"
format_stamp: "Formatted at 2026-09-23 17:28:35 on dist-test-slave-p7jt"
server_key: "311fcdf2ef8efe907148f46422cc5abb"
server_key_iv: "1d5ea28ea2ea7831093d44c43f2eab86"
server_key_version: "encryptionkey@0"
I20260923 17:28:35.409224 12413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:35.446689 12413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:35.450870 12413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:35.451133 12413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:35.468046 12413 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:40381
I20260923 17:28:35.468019 12478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:40381 every 8 connection(s)
I20260923 17:28:35.468935 12413 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:35.472574 12479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:35.475265 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 12413
I20260923 17:28:35.475453 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:35.476184 8311 external_mini_cluster.cc:1428] Setting key 1b35e7d8c5a4d4ba5b62de4e08e67091
I20260923 17:28:35.485287 12479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046: Bootstrap starting.
I20260923 17:28:35.489022 12479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:35.490034 12479 log.cc:824] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:28:35 dist-test-slave-p7jt krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:35.493288 12479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046: No bootstrap required, opened a new log
I20260923 17:28:35.500777 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:35.478517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43174 (local address 127.8.29.254:40381)
0923 17:28:35.479326 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:35.479340 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:35.479386 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:28:35.481132 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:35.481157 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:35.481649 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:35.482037 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:35.483958 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:35.485286 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:35.486735 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:35.487843 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:35.488033 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:35.492274 (+ 4241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:35.492284 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:35.492305 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:35.492350 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:35.495796 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:35.496780 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:35.496797 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:35.496814 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:35.497075 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:35.497506 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:35.497510 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:35.497512 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:35.498233 (+ 721us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:35.498518 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:35.499072 (+ 554us) server_negotiation.cc:299] Negotiation successful
0923 17:28:35.499544 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":384,"threads_started":1}
I20260923 17:28:35.502080 12479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604475cc9ce2485f84789280b45f9046" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40381 } }
I20260923 17:28:35.502576 12479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:35.502694 12479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 604475cc9ce2485f84789280b45f9046, State: Initialized, Role: FOLLOWER
I20260923 17:28:35.503372 12479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604475cc9ce2485f84789280b45f9046" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40381 } }
I20260923 17:28:35.503644 12479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:35.503784 12479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:35.503952 12479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:35.506191 12479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604475cc9ce2485f84789280b45f9046" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40381 } }
I20260923 17:28:35.506760 12479 leader_election.cc:302] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 604475cc9ce2485f84789280b45f9046; no voters:
I20260923 17:28:35.507344 12479 leader_election.cc:288] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:35.507507 12484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:35.508482 12484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [term 1 LEADER]: Becoming Leader. State: Replica: 604475cc9ce2485f84789280b45f9046, State: Running, Role: LEADER
I20260923 17:28:35.509125 12484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604475cc9ce2485f84789280b45f9046" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40381 } }
I20260923 17:28:35.509495 12479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:35.511778 12486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 604475cc9ce2485f84789280b45f9046. Latest consensus state: current_term: 1 leader_uuid: "604475cc9ce2485f84789280b45f9046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604475cc9ce2485f84789280b45f9046" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40381 } } }
I20260923 17:28:35.512200 12486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:35.512017 12485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "604475cc9ce2485f84789280b45f9046" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "604475cc9ce2485f84789280b45f9046" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40381 } } }
I20260923 17:28:35.512306 12485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:35.515017 12493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:35.520334 12493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:35.530830 12493 catalog_manager.cc:1347] Generated new cluster ID: 10e4f29a6dd747c6b6705d014a310b54
I20260923 17:28:35.530956 12493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:35.569957 12493 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:35.571931 12493 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:35.596565 12493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046: Generated new TSK 0
I20260923 17:28:35.597740 12493 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:28:35.654776 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40381
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:35.980938 12505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:35.981228 12505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:35.981355 12505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:35.988876 12505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:35.989009 12505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:35.989161 12505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:35.999251 12505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:36.000643 12505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:36.001972 12505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:36.015007 12511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:36.014963 12513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:36.014900 12510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:36.015686 12505 server_base.cc:1034] running on GCE node
I20260923 17:28:36.016335 12505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:36.017329 12505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:36.018570 12505 hybrid_clock.cc:648] HybridClock initialized: now 1790184516018524 us; error 67 us; skew 500 ppm
I20260923 17:28:36.018922 12505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:36 dist-test-slave-p7jt krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184516, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:36.022022 12505 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:28:36.023200 12505 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:36.023293 12505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:36.023574 12505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:36.026558 12505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "a481b22cff5045b5983945592b610076"
format_stamp: "Formatted at 2026-09-23 17:28:36 on dist-test-slave-p7jt"
server_key: "2682eb13151ae721a2ed748b4096acc4"
server_key_iv: "2c20d509d3537214e6444ea41e99554e"
server_key_version: "encryptionkey@0"
I20260923 17:28:36.027436 12505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "a481b22cff5045b5983945592b610076"
format_stamp: "Formatted at 2026-09-23 17:28:36 on dist-test-slave-p7jt"
server_key: "2682eb13151ae721a2ed748b4096acc4"
server_key_iv: "2c20d509d3537214e6444ea41e99554e"
server_key_version: "encryptionkey@0"
I20260923 17:28:36.032711 12505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 17:28:36.036353 12519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:36.037811 12505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:36.037974 12505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "a481b22cff5045b5983945592b610076"
format_stamp: "Formatted at 2026-09-23 17:28:36 on dist-test-slave-p7jt"
server_key: "2682eb13151ae721a2ed748b4096acc4"
server_key_iv: "2c20d509d3537214e6444ea41e99554e"
server_key_version: "encryptionkey@0"
I20260923 17:28:36.038216 12505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:36.049834 12505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:36.053792 12505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:36.054044 12505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:36.055117 12505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:36.056535 12505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:36.056629 12505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:36.056707 12505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:36.056780 12505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:36.086791 12505 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:33627
I20260923 17:28:36.086843 12631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:33627 every 8 connection(s)
I20260923 17:28:36.087646 12505 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:36.095521 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 12505
I20260923 17:28:36.095728 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:36.096386 8311 external_mini_cluster.cc:1428] Setting key 0ca8c1393f30cd0b88c75ea16abc86ee
Sep 23 17:28:36 dist-test-slave-p7jt krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184516, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:36.109933 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:36.089983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:48115 (local address 127.8.29.254:40381)
0923 17:28:36.090408 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:36.090412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:36.091424 (+ 1012us) server_negotiation.cc:407] Connection header received
0923 17:28:36.093240 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:36.093245 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:36.093320 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:36.093421 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:36.095069 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:36.096049 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:36.098223 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:36.098775 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:36.098882 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:36.103288 (+ 4406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:36.103294 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:36.103297 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:36.103336 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:36.106223 (+ 2887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:36.107434 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:36.107440 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:36.107442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:36.107607 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:36.108102 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:36.108105 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:36.108107 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:36.108396 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:36.108510 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:36.109461 (+ 951us) server_negotiation.cc:299] Negotiation successful
0923 17:28:36.109668 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":212,"threads_started":1}
I20260923 17:28:36.111346 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:36.090559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:40381 (local address 127.8.29.193:48115)
0923 17:28:36.091249 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:36.091293 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:36.092927 (+ 1634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:36.093654 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:36.093664 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:36.094210 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:36.094874 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:36.094894 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:36.096276 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:36.096282 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:36.098046 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:36.098061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:36.099019 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:36.099027 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:36.099365 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:36.100109 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:36.100143 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:36.102961 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:36.106432 (+ 3471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:36.106442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:36.106497 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:36.107256 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:36.107731 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:36.107735 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:36.107739 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:36.108000 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:36.108514 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:36.108522 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:36.109004 (+ 482us) client_negotiation.cc:769] Sending connection context
0923 17:28:36.109337 (+ 333us) client_negotiation.cc:240] Negotiation successful
0923 17:28:36.109891 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":248,"threads_started":1}
I20260923 17:28:36.113165 12632 heartbeater.cc:347] Connected to a master server at 127.8.29.254:40381
I20260923 17:28:36.113471 12632 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:36.114153 12632 heartbeater.cc:511] Master 127.8.29.254:40381 requested a full tablet report, sending...
I20260923 17:28:36.116500 12444 ts_manager.cc:194] Registered new tserver with Master: a481b22cff5045b5983945592b610076 (127.8.29.193:33627)
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:28:36.118120 12444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:48115
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:28:36.143014 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40381
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:36.469304 12638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:36.469635 12638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:36.469758 12638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:36.477123 12638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:36.477227 12638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:36.477375 12638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:36.487314 12638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:36.488646 12638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:36.490018 12638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:36.503254 12646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:36.503186 12644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:36.503283 12643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:36.503705 12638 server_base.cc:1034] running on GCE node
I20260923 17:28:36.504523 12638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:36.505519 12638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:36.506732 12638 hybrid_clock.cc:648] HybridClock initialized: now 1790184516506660 us; error 63 us; skew 500 ppm
I20260923 17:28:36.507097 12638 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:36 dist-test-slave-p7jt krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184516, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:36.510267 12638 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:28:36.511459 12638 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:36.511565 12638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:36.511885 12638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:36.515157 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "8bb0c0db21d84f4986b4dc19de160d35"
format_stamp: "Formatted at 2026-09-23 17:28:36 on dist-test-slave-p7jt"
server_key: "f3585f5d41135272f2829b98524cb8f3"
server_key_iv: "2d8dd34c23757818c6760a22dcc90364"
server_key_version: "encryptionkey@0"
I20260923 17:28:36.516006 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "8bb0c0db21d84f4986b4dc19de160d35"
format_stamp: "Formatted at 2026-09-23 17:28:36 on dist-test-slave-p7jt"
server_key: "f3585f5d41135272f2829b98524cb8f3"
server_key_iv: "2d8dd34c23757818c6760a22dcc90364"
server_key_version: "encryptionkey@0"
I20260923 17:28:36.523778 12638 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260923 17:28:36.528529 12652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:36.530174 12638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:28:36.530352 12638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "8bb0c0db21d84f4986b4dc19de160d35"
format_stamp: "Formatted at 2026-09-23 17:28:36 on dist-test-slave-p7jt"
server_key: "f3585f5d41135272f2829b98524cb8f3"
server_key_iv: "2d8dd34c23757818c6760a22dcc90364"
server_key_version: "encryptionkey@0"
I20260923 17:28:36.530838 12638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:36.548524 12638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:36.552829 12638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:36.553131 12638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:36.554190 12638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:36.555614 12638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:36.555719 12638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:36.555824 12638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:36.555895 12638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:36.586045 12638 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:38907
I20260923 17:28:36.586100 12764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:38907 every 8 connection(s)
I20260923 17:28:36.586928 12638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:36.592156 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 12638
I20260923 17:28:36.592329 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:36.592875 8311 external_mini_cluster.cc:1428] Setting key d97275776b397858d8a8b1b2786692d9
Sep 23 17:28:36 dist-test-slave-p7jt krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184516, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:36.606834 12634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:36.589537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:48997 (local address 127.8.29.254:40381)
0923 17:28:36.589763 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:36.589771 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:36.590944 (+ 1173us) server_negotiation.cc:407] Connection header received
0923 17:28:36.592746 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:36.592752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:36.592858 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:36.593002 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:36.594612 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:36.595314 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:36.596967 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:36.597552 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:36.597670 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:36.601144 (+ 3474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:36.601149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:36.601153 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:36.601186 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:36.603636 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:36.604495 (+ 859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:36.604498 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:36.604500 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:36.604627 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:36.605080 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:36.605084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:36.605086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:36.605351 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:36.605476 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:36.606332 (+ 856us) server_negotiation.cc:299] Negotiation successful
0923 17:28:36.606602 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:28:36.608318 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:36.590027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:40381 (local address 127.8.29.194:48997)
0923 17:28:36.590776 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:36.590822 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:36.592459 (+ 1637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:36.593225 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:36.593234 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:36.593791 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:36.594415 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:36.594438 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:36.595463 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:36.595467 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:36.596782 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:36.596797 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:36.597764 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:36.597768 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:36.597965 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:36.598614 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:36.598637 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:36.600896 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:36.603775 (+ 2879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:36.603785 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:36.603803 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:36.604388 (+ 585us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:36.604728 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:36.604732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:36.604736 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:36.604956 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:36.605469 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:36.605478 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:36.605918 (+ 440us) client_negotiation.cc:769] Sending connection context
0923 17:28:36.606235 (+ 317us) client_negotiation.cc:240] Negotiation successful
0923 17:28:36.606849 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":279,"threads_started":1}
I20260923 17:28:36.609862 12765 heartbeater.cc:347] Connected to a master server at 127.8.29.254:40381
I20260923 17:28:36.610226 12765 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
I20260923 17:28:36.611021 12765 heartbeater.cc:511] Master 127.8.29.254:40381 requested a full tablet report, sending...
I20260923 17:28:36.612555 12444 ts_manager.cc:194] Registered new tserver with Master: 8bb0c0db21d84f4986b4dc19de160d35 (127.8.29.194:38907)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
I20260923 17:28:36.613504 12444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:48997
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:28:36.638626 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40381
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:44361
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:36.960115 12770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:36.960415 12770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:36.960515 12770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:36.967785 12770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:36.967905 12770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:36.968021 12770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:36.977979 12770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:44361
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40381
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:36.979374 12770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:36.980628 12770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:36.993428 12778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:36.993502 12775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:36.993534 12776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:36.994431 12770 server_base.cc:1034] running on GCE node
I20260923 17:28:36.995133 12770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:36.996197 12770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:36.997418 12770 hybrid_clock.cc:648] HybridClock initialized: now 1790184516997341 us; error 66 us; skew 500 ppm
I20260923 17:28:36.997754 12770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:37 dist-test-slave-p7jt krb5kdc[12400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184517, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:37.011951 12770 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:28:37.013341 12770 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:37.013461 12770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:37.013793 12770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:37.017169 12770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "15e9b15f22ff46349bf7d24a393696af"
format_stamp: "Formatted at 2026-09-23 17:28:37 on dist-test-slave-p7jt"
server_key: "0f59f1d54e5f4e201d118848566e5df1"
server_key_iv: "bb7fd791e302a88fc76b3e238341d34a"
server_key_version: "encryptionkey@0"
I20260923 17:28:37.018137 12770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "15e9b15f22ff46349bf7d24a393696af"
format_stamp: "Formatted at 2026-09-23 17:28:37 on dist-test-slave-p7jt"
server_key: "0f59f1d54e5f4e201d118848566e5df1"
server_key_iv: "bb7fd791e302a88fc76b3e238341d34a"
server_key_version: "encryptionkey@0"
I20260923 17:28:37.023867 12770 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260923 17:28:37.027673 12784 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:37.029145 12770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:37.029315 12770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "15e9b15f22ff46349bf7d24a393696af"
format_stamp: "Formatted at 2026-09-23 17:28:37 on dist-test-slave-p7jt"
server_key: "0f59f1d54e5f4e201d118848566e5df1"
server_key_iv: "bb7fd791e302a88fc76b3e238341d34a"
server_key_version: "encryptionkey@0"
I20260923 17:28:37.029533 12770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:37.068392 12770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:37.072847 12770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:37.073109 12770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:37.074101 12770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:37.075516 12770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:37.075614 12770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:37.075719 12770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:37.075786 12770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:37.108126 12770 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:39997
I20260923 17:28:37.108175 12896 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:39997 every 8 connection(s)
I20260923 17:28:37.109036 12770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:37.117969 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 12770
I20260923 17:28:37.118150 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:37.118667 8311 external_mini_cluster.cc:1428] Setting key 2573dbff6475640a373ba2627c4477db
I20260923 17:28:37.121537 12632 heartbeater.cc:503] Master 127.8.29.254:40381 was elected leader, sending a full tablet report...
Sep 23 17:28:37 dist-test-slave-p7jt krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184517, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:37.131775 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.111568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:41733 (local address 127.8.29.254:40381)
0923 17:28:37.111987 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.111992 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.112883 (+ 891us) server_negotiation.cc:407] Connection header received
0923 17:28:37.114866 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.114871 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.114985 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.115085 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:37.116750 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.117825 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.119747 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.120376 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.120496 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.124659 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:37.124665 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:37.124668 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:37.124704 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:37.127828 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.129151 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.129156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.129159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.129322 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.129920 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.129924 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.129926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.130266 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:37.130395 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.131305 (+ 910us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.131503 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":214,"threads_started":1}
I20260923 17:28:37.132899 12900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.111976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:40381 (local address 127.8.29.195:41733)
0923 17:28:37.112715 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:37.112775 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:37.114591 (+ 1816us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:37.115356 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:37.115365 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:37.115931 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:37.116551 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.116570 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.118047 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.118052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.119549 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.119567 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.120661 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.120671 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.120944 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.121673 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:37.121706 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:37.124350 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:37.128143 (+ 3793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:37.128157 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:37.128183 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:37.128956 (+ 773us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:37.129501 (+ 545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:37.129506 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:37.129509 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:37.129782 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:37.130419 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:37.130427 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:37.130896 (+ 469us) client_negotiation.cc:769] Sending connection context
0923 17:28:37.131149 (+ 253us) client_negotiation.cc:240] Negotiation successful
0923 17:28:37.131583 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":295,"threads_started":1}
I20260923 17:28:37.134721 12897 heartbeater.cc:347] Connected to a master server at 127.8.29.254:40381
I20260923 17:28:37.135149 12897 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:37.135866 12897 heartbeater.cc:511] Master 127.8.29.254:40381 requested a full tablet report, sending...
I20260923 17:28:37.137765 12444 ts_manager.cc:194] Registered new tserver with Master: 15e9b15f22ff46349bf7d24a393696af (127.8.29.195:39997)
I20260923 17:28:37.138682 12444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:41733
I20260923 17:28:37.147593 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:37.164052 12899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.152194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.8.29.254:40381)
0923 17:28:37.152432 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.152439 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.152803 (+ 364us) server_negotiation.cc:407] Connection header received
0923 17:28:37.152943 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.152947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.153014 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.153101 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:37.154518 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.155243 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.156515 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.156945 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.157043 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.159288 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:37.159293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:37.159296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:37.159334 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:37.161773 (+ 2439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.162363 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.162366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.162369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.162532 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.162822 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.162825 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.162827 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.163160 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:37.163341 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.163573 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.163766 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:28:37.168066 12444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:37.175204 12444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:37.198051 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.188236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.8.29.193:33627)
0923 17:28:37.188765 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.188773 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.188801 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:28:37.188892 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.188899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.189164 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.189380 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:37.190275 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.192225 (+ 1950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.195494 (+ 3269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.196640 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.196755 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.197017 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.197489 (+ 472us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.197684 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":1128,"threads_started":1}
I20260923 17:28:37.198637 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.187712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:33627 (local address 127.0.0.1:58290)
0923 17:28:37.188505 (+ 793us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:37.188549 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:37.188688 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:37.189476 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:37.189484 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:37.189521 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:37.190014 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.190034 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.192472 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.192479 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.195260 (+ 2781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.195277 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.196991 (+ 1714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.196998 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.197219 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.197275 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 17:28:37.197408 (+ 133us) client_negotiation.cc:240] Negotiation successful
0923 17:28:37.197609 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":672,"threads_started":1}
I20260923 17:28:37.202881 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 56a2a9ee39ec408780dc054fdc81203b (DEFAULT_TABLE table=test-table [id=effaedd1e0c640c283db9f1b0857b17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:37.203585 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.189243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:39997 (local address 127.0.0.1:60154)
0923 17:28:37.190046 (+ 803us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:37.190064 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:37.190170 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:37.196624 (+ 6454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:37.196629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:37.196658 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:37.197136 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.197148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.199544 (+ 2396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.199549 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.201248 (+ 1699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.201259 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.202892 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.202899 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.203059 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.203093 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:28:37.203157 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 17:28:37.203255 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":657,"mutex_wait_us":1,"thread_start_us":508,"threads_started":1}
I20260923 17:28:37.203732 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.190552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:38907 (local address 127.0.0.1:34548)
0923 17:28:37.194967 (+ 4415us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:37.194990 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:37.195188 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:37.196154 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:37.196159 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:37.196184 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:37.196581 (+ 397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.196595 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.198334 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.198338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.200494 (+ 2156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.200514 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.202741 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.202748 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.202970 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.203011 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 17:28:37.203117 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 17:28:37.203274 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4299,"mutex_wait_us":1,"thread_start_us":4156,"threads_started":1}
I20260923 17:28:37.204479 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.190307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34548 (local address 127.8.29.194:38907)
0923 17:28:37.190774 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.190783 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.195080 (+ 4297us) server_negotiation.cc:407] Connection header received
0923 17:28:37.195387 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.195397 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.195703 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.195881 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:37.196765 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.198180 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.200686 (+ 2506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.202148 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.202325 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.202573 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.203973 (+ 1400us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.204162 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":342,"threads_started":1}
I20260923 17:28:37.204617 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a2a9ee39ec408780dc054fdc81203b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:37.204617 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.189630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60154 (local address 127.8.29.195:39997)
0923 17:28:37.196012 (+ 6382us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.196021 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.196059 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:28:37.196155 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.196162 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.196398 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.196594 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:37.197353 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.199360 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.201406 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.202661 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.203957 (+ 1296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.204166 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.204222 (+ 56us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.204340 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5754,"thread_start_us":5943,"threads_started":1}
I20260923 17:28:37.208050 12832 tablet_service.cc:1467] Processing CreateTablet for tablet 56a2a9ee39ec408780dc054fdc81203b (DEFAULT_TABLE table=test-table [id=effaedd1e0c640c283db9f1b0857b17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:37.208091 12700 tablet_service.cc:1467] Processing CreateTablet for tablet 56a2a9ee39ec408780dc054fdc81203b (DEFAULT_TABLE table=test-table [id=effaedd1e0c640c283db9f1b0857b17e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:37.209278 12832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a2a9ee39ec408780dc054fdc81203b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:37.209472 12700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56a2a9ee39ec408780dc054fdc81203b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:37.221837 12916 tablet_bootstrap.cc:492] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: Bootstrap starting.
I20260923 17:28:37.224932 12918 tablet_bootstrap.cc:492] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: Bootstrap starting.
I20260923 17:28:37.226475 12916 tablet_bootstrap.cc:654] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:37.227837 12917 tablet_bootstrap.cc:492] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: Bootstrap starting.
I20260923 17:28:37.228008 12916 log.cc:824] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:37.228627 12918 tablet_bootstrap.cc:654] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:37.229682 12918 log.cc:824] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:37.231150 12917 tablet_bootstrap.cc:654] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:37.231432 12916 tablet_bootstrap.cc:492] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: No bootstrap required, opened a new log
I20260923 17:28:37.231729 12916 ts_tablet_manager.cc:1397] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260923 17:28:37.232124 12917 log.cc:824] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:37.232743 12918 tablet_bootstrap.cc:492] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: No bootstrap required, opened a new log
I20260923 17:28:37.233016 12918 ts_tablet_manager.cc:1397] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260923 17:28:37.236392 12917 tablet_bootstrap.cc:492] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: No bootstrap required, opened a new log
I20260923 17:28:37.236680 12917 ts_tablet_manager.cc:1397] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260923 17:28:37.242982 12916 raft_consensus.cc:348] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } }
I20260923 17:28:37.243531 12916 raft_consensus.cc:374] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:37.243630 12916 raft_consensus.cc:729] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e9b15f22ff46349bf7d24a393696af, State: Initialized, Role: FOLLOWER
I20260923 17:28:37.243412 12918 raft_consensus.cc:348] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } }
I20260923 17:28:37.243898 12918 raft_consensus.cc:374] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:37.243991 12918 raft_consensus.cc:729] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bb0c0db21d84f4986b4dc19de160d35, State: Initialized, Role: FOLLOWER
I20260923 17:28:37.243868 12917 raft_consensus.cc:348] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } }
I20260923 17:28:37.244284 12917 raft_consensus.cc:374] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:37.244375 12917 raft_consensus.cc:729] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a481b22cff5045b5983945592b610076, State: Initialized, Role: FOLLOWER
I20260923 17:28:37.244351 12916 consensus_queue.cc:260] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } }
I20260923 17:28:37.244663 12918 consensus_queue.cc:260] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } }
I20260923 17:28:37.244913 12917 consensus_queue.cc:260] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } }
I20260923 17:28:37.246657 12918 ts_tablet_manager.cc:1428] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260923 17:28:37.247697 12765 heartbeater.cc:503] Master 127.8.29.254:40381 was elected leader, sending a full tablet report...
I20260923 17:28:37.247958 12897 heartbeater.cc:503] Master 127.8.29.254:40381 was elected leader, sending a full tablet report...
I20260923 17:28:37.250346 12917 ts_tablet_manager.cc:1428] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: Time spent starting tablet: real 0.014s user 0.005s sys 0.008s
I20260923 17:28:37.251536 12916 ts_tablet_manager.cc:1428] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: Time spent starting tablet: real 0.020s user 0.010s sys 0.007s
I20260923 17:28:37.296048 12923 raft_consensus.cc:482] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:37.296387 12923 raft_consensus.cc:504] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } }
I20260923 17:28:37.298120 12923 leader_election.cc:288] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a481b22cff5045b5983945592b610076 (127.8.29.193:33627), 15e9b15f22ff46349bf7d24a393696af (127.8.29.195:39997)
I20260923 17:28:37.305073 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.298698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:47917 (local address 127.8.29.193:33627)
0923 17:28:37.298868 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.298873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.299241 (+ 368us) server_negotiation.cc:407] Connection header received
0923 17:28:37.299517 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.299523 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.299601 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.299668 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:37.300306 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.301171 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.303173 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.304042 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.304172 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.304226 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.304633 (+ 407us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.304773 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:28:37.305087 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.298411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:33627 (local address 127.8.29.194:47917)
0923 17:28:37.299142 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:37.299161 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:37.299313 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:37.299870 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:37.299873 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:37.299895 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:37.300172 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.300180 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.301330 (+ 1150us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.301336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.303009 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.303024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.304264 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.304270 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.304438 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.304473 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:28:37.304567 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:28:37.304673 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"mutex_wait_us":25,"thread_start_us":312,"threads_started":1}
I20260923 17:28:37.305183 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.299404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:50803 (local address 127.8.29.195:39997)
0923 17:28:37.299605 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.299610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.299775 (+ 165us) server_negotiation.cc:407] Connection header received
0923 17:28:37.299935 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.299940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.300008 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.300079 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:37.300706 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.301551 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.303436 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.304246 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.304403 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.304471 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.304817 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.304950 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:28:37.305823 12926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.299165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:39997 (local address 127.8.29.194:50803)
0923 17:28:37.299676 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:37.299690 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:37.299815 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:37.300112 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:37.300117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:37.300145 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:37.300556 (+ 411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.300568 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.301707 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.301711 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.303266 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:37.303280 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.304468 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.304473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:37.304645 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.304675 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:28:37.304769 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:28:37.304899 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":21,"thread_start_us":439,"threads_started":1}
I20260923 17:28:37.306159 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a2a9ee39ec408780dc054fdc81203b" candidate_uuid: "8bb0c0db21d84f4986b4dc19de160d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a481b22cff5045b5983945592b610076" is_pre_election: true
I20260923 17:28:37.306699 12587 raft_consensus.cc:2454] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bb0c0db21d84f4986b4dc19de160d35 in term 0.
I20260923 17:28:37.306762 12852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a2a9ee39ec408780dc054fdc81203b" candidate_uuid: "8bb0c0db21d84f4986b4dc19de160d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e9b15f22ff46349bf7d24a393696af" is_pre_election: true
I20260923 17:28:37.307287 12852 raft_consensus.cc:2454] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bb0c0db21d84f4986b4dc19de160d35 in term 0.
I20260923 17:28:37.307318 12653 leader_election.cc:302] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bb0c0db21d84f4986b4dc19de160d35, a481b22cff5045b5983945592b610076; no voters:
I20260923 17:28:37.307729 12923 raft_consensus.cc:2781] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:37.307888 12923 raft_consensus.cc:482] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:37.307982 12923 raft_consensus.cc:3037] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:37.310235 12923 raft_consensus.cc:504] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } }
I20260923 17:28:37.310870 12923 leader_election.cc:288] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [CANDIDATE]: Term 1 election: Requested vote from peers a481b22cff5045b5983945592b610076 (127.8.29.193:33627), 15e9b15f22ff46349bf7d24a393696af (127.8.29.195:39997)
I20260923 17:28:37.311404 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a2a9ee39ec408780dc054fdc81203b" candidate_uuid: "8bb0c0db21d84f4986b4dc19de160d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a481b22cff5045b5983945592b610076"
I20260923 17:28:37.311522 12852 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56a2a9ee39ec408780dc054fdc81203b" candidate_uuid: "8bb0c0db21d84f4986b4dc19de160d35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e9b15f22ff46349bf7d24a393696af"
I20260923 17:28:37.311605 12587 raft_consensus.cc:3037] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:37.311728 12852 raft_consensus.cc:3037] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:37.313999 12852 raft_consensus.cc:2454] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bb0c0db21d84f4986b4dc19de160d35 in term 1.
I20260923 17:28:37.314052 12587 raft_consensus.cc:2454] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bb0c0db21d84f4986b4dc19de160d35 in term 1.
I20260923 17:28:37.314448 12653 leader_election.cc:302] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15e9b15f22ff46349bf7d24a393696af, 8bb0c0db21d84f4986b4dc19de160d35; no voters:
I20260923 17:28:37.314788 12923 raft_consensus.cc:2781] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:37.315232 12923 raft_consensus.cc:686] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 1 LEADER]: Becoming Leader. State: Replica: 8bb0c0db21d84f4986b4dc19de160d35, State: Running, Role: LEADER
I20260923 17:28:37.315755 12923 consensus_queue.cc:237] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } }
I20260923 17:28:37.321156 12443 catalog_manager.cc:5495] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8bb0c0db21d84f4986b4dc19de160d35 (127.8.29.194). New cstate: current_term: 1 leader_uuid: "8bb0c0db21d84f4986b4dc19de160d35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8bb0c0db21d84f4986b4dc19de160d35" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 38907 } health_report { overall_health: HEALTHY } } }
W20260923 17:28:37.338835 12766 tablet.cc:2369] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:28:37.339718 12633 tablet.cc:2369] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:28:37.360517 12898 tablet.cc:2369] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:37.363494 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.357941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34556 (local address 127.8.29.194:38907)
0923 17:28:37.358149 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.358154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.358171 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:28:37.358260 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.358264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.358340 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.358513 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:37.358830 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.359534 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.360983 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.361556 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.361665 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.362276 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:37.362861 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:37.363006 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.363113 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.363201 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:28:37.372052 12587 raft_consensus.cc:1261] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [term 1 FOLLOWER]: Refusing update from remote peer 8bb0c0db21d84f4986b4dc19de160d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:37.372052 12852 raft_consensus.cc:1261] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [term 1 FOLLOWER]: Refusing update from remote peer 8bb0c0db21d84f4986b4dc19de160d35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:37.373000 12927 consensus_queue.cc:1035] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e9b15f22ff46349bf7d24a393696af" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 39997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:37.373395 12923 consensus_queue.cc:1035] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a481b22cff5045b5983945592b610076" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 33627 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:37.384974 12930 mvcc.cc:205] Tried to move back new op lower bound from 7332595783145922560 to 7332595782929653760. Current Snapshot: MvccSnapshot[applied={T|T < 7332595783145922560}]
I20260923 17:28:37.399322 12443 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:37.399585 12443 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:37.403064 12443 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 604475cc9ce2485f84789280b45f9046: Sending DeleteTablet for 3 replicas of tablet 56a2a9ee39ec408780dc054fdc81203b
I20260923 17:28:37.404263 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a2a9ee39ec408780dc054fdc81203b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:37 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:58290
I20260923 17:28:37.404384 12700 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a2a9ee39ec408780dc054fdc81203b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:37 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:34548
I20260923 17:28:37.404384 12832 tablet_service.cc:1514] Processing DeleteTablet for tablet 56a2a9ee39ec408780dc054fdc81203b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:37 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:60154
I20260923 17:28:37.405838 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 12505
I20260923 17:28:37.408891 12941 tablet_replica.cc:330] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: stopping tablet replica
I20260923 17:28:37.409433 12941 raft_consensus.cc:2229] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:37.410027 12941 raft_consensus.cc:2258] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:37.410900 12940 tablet_replica.cc:330] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: stopping tablet replica
I20260923 17:28:37.411367 12940 raft_consensus.cc:2229] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:37.412060 12940 raft_consensus.cc:2258] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:37.412916 12939 tablet_replica.cc:330] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: stopping tablet replica
I20260923 17:28:37.413363 12939 raft_consensus.cc:2229] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:37.413959 12939 raft_consensus.cc:2258] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:37.414803 12941 ts_tablet_manager.cc:1905] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:37.416256 12939 ts_tablet_manager.cc:1905] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:37.416270 12940 ts_tablet_manager.cc:1905] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:37.421846 12941 ts_tablet_manager.cc:1918] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:37.422052 12941 log.cc:1196] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/56a2a9ee39ec408780dc054fdc81203b
I20260923 17:28:37.422588 12941 ts_tablet_manager.cc:1939] T 56a2a9ee39ec408780dc054fdc81203b P 15e9b15f22ff46349bf7d24a393696af: Deleting consensus metadata
I20260923 17:28:37.423015 12939 ts_tablet_manager.cc:1918] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:37.423184 12939 log.cc:1196] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/56a2a9ee39ec408780dc054fdc81203b
I20260923 17:28:37.423676 12939 ts_tablet_manager.cc:1939] T 56a2a9ee39ec408780dc054fdc81203b P a481b22cff5045b5983945592b610076: Deleting consensus metadata
I20260923 17:28:37.424638 12428 catalog_manager.cc:4841] TS 15e9b15f22ff46349bf7d24a393696af (127.8.29.195:39997): tablet 56a2a9ee39ec408780dc054fdc81203b (table test-table [id=effaedd1e0c640c283db9f1b0857b17e]) successfully deleted
I20260923 17:28:37.425355 12428 catalog_manager.cc:4841] TS a481b22cff5045b5983945592b610076 (127.8.29.193:33627): tablet 56a2a9ee39ec408780dc054fdc81203b (table test-table [id=effaedd1e0c640c283db9f1b0857b17e]) successfully deleted
Sep 23 17:28:37 dist-test-slave-p7jt krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:37.428335 12940 ts_tablet_manager.cc:1918] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:37.428509 12940 log.cc:1196] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/56a2a9ee39ec408780dc054fdc81203b
I20260923 17:28:37.429081 12940 ts_tablet_manager.cc:1939] T 56a2a9ee39ec408780dc054fdc81203b P 8bb0c0db21d84f4986b4dc19de160d35: Deleting consensus metadata
I20260923 17:28:37.430944 12428 catalog_manager.cc:4841] TS 8bb0c0db21d84f4986b4dc19de160d35 (127.8.29.194:38907): tablet 56a2a9ee39ec408780dc054fdc81203b (table test-table [id=effaedd1e0c640c283db9f1b0857b17e]) successfully deleted
I20260923 17:28:37.433939 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.406770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58300 (local address 127.8.29.193:33627)
0923 17:28:37.406920 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.406927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.408162 (+ 1235us) server_negotiation.cc:407] Connection header received
0923 17:28:37.408367 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.408373 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.408493 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.408597 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:37.410051 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.411255 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.416516 (+ 5261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.417223 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.417341 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.427064 (+ 9723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:37.427079 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:37.427098 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:37.427147 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:37.430884 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.431802 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.431810 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.431812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.431953 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.432434 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.432438 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.432440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.432856 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:37.433138 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.433450 (+ 312us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.433635 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 17:28:37.434737 12627 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:37.525038 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 12505
I20260923 17:28:37.541294 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 12638
Sep 23 17:28:37 dist-test-slave-p7jt krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:28:37.559901 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.542159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.8.29.194:38907)
0923 17:28:37.542344 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.542349 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.542522 (+ 173us) server_negotiation.cc:407] Connection header received
0923 17:28:37.542722 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.542726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.542797 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.542900 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:37.544474 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.545343 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.547274 (+ 1931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.547727 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.547836 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.552773 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:37.552790 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:37.552808 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:37.552856 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:37.556065 (+ 3209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.557135 (+ 1070us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.557147 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.557150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.557375 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.557919 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.557924 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.557926 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.558368 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:37.558793 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.559370 (+ 577us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.559598 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:28:37.560747 12760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:37.670104 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 12638
I20260923 17:28:37.685262 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 12770
Sep 23 17:28:37 dist-test-slave-p7jt krb5kdc[12400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184514, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:28:37.700518 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:37.686045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60158 (local address 127.8.29.195:39997)
0923 17:28:37.686226 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:37.686231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:37.686250 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:28:37.686341 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:37.686345 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:37.686419 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:37.686572 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:37.687852 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.688614 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.689950 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:37.690372 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:37.690509 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:37.694979 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:37.694997 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:37.695013 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:37.695060 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:37.697833 (+ 2773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.698639 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.698648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.698650 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.698799 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:37.699152 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:37.699155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:37.699157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:37.699524 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:37.699738 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:37.700056 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 17:28:37.700261 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:28:37.701217 12892 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:37.793663 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 12770
I20260923 17:28:37.806862 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 12413
I20260923 17:28:37.808195 12474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:37.915879 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 12413
2026-09-23T17:28:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4089 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T17:28:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:37Z Disabled control of system clock
I20260923 17:28:37.977365 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:36395
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:36395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:28:38.301343 12950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:38.301637 12950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:38.301719 12950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:38.308944 12950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:38.309064 12950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:38.309100 12950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:38.309155 12950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:38.309198 12950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:38.318419 12950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:36395
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:36395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:38.319710 12950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:38.321506 12950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:38.333489 12956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:38.333578 12955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:38.333609 12958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:38.334211 12950 server_base.cc:1034] running on GCE node
I20260923 17:28:38.335191 12950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:38.336819 12950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:38.338104 12950 hybrid_clock.cc:648] HybridClock initialized: now 1790184518338043 us; error 51 us; skew 500 ppm
I20260923 17:28:38.338572 12950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:28:38.340943 12950 webserver.cc:460] Webserver started at http://127.8.29.254:40523/ using document root <none> and password file <none>
I20260923 17:28:38.341676 12950 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:38.341784 12950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:38.342110 12950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:38.345182 12950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "6a9899377b4a405ab48c9e6c5a19b37f"
format_stamp: "Formatted at 2026-09-23 17:28:38 on dist-test-slave-p7jt"
server_key: "f706cf90289fd35320280e012f3290c3"
server_key_iv: "27acaa9174b396849541438e558d51b0"
server_key_version: "encryptionkey@0"
I20260923 17:28:38.346006 12950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "6a9899377b4a405ab48c9e6c5a19b37f"
format_stamp: "Formatted at 2026-09-23 17:28:38 on dist-test-slave-p7jt"
server_key: "f706cf90289fd35320280e012f3290c3"
server_key_iv: "27acaa9174b396849541438e558d51b0"
server_key_version: "encryptionkey@0"
I20260923 17:28:38.353223 12950 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260923 17:28:38.357295 12964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:38.358817 12950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:38.358999 12950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "6a9899377b4a405ab48c9e6c5a19b37f"
format_stamp: "Formatted at 2026-09-23 17:28:38 on dist-test-slave-p7jt"
server_key: "f706cf90289fd35320280e012f3290c3"
server_key_iv: "27acaa9174b396849541438e558d51b0"
server_key_version: "encryptionkey@0"
I20260923 17:28:38.359220 12950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:38.365260 12950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:38.366083 12950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:38.366353 12950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:38.383519 12950 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:36395
I20260923 17:28:38.383563 13015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:36395 every 8 connection(s)
I20260923 17:28:38.384881 12950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:38.385866 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 12950
I20260923 17:28:38.386085 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:38.386915 8311 external_mini_cluster.cc:1428] Setting key dd2ce5ba02b5f9790a02242b0518bae9
I20260923 17:28:38.389735 13016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:38.395412 13019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:38.389095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.8.29.254:36395)
0923 17:28:38.390136 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:38.390148 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:38.390191 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:28:38.391791 (+ 1600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:38.391815 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:38.392252 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:38.392666 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:38.393717 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:38.393724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:38.393737 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:38.393751 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:38.393826 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:38.393843 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:38.394069 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:38.394207 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:28:38.394451 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":344,"threads_started":1}
I20260923 17:28:38.401054 13016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f: Bootstrap starting.
I20260923 17:28:38.404146 13016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:38.405262 13016 log.cc:824] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:38.408200 13016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f: No bootstrap required, opened a new log
I20260923 17:28:38.415938 13016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9899377b4a405ab48c9e6c5a19b37f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36395 } }
I20260923 17:28:38.416361 13016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:38.416455 13016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9899377b4a405ab48c9e6c5a19b37f, State: Initialized, Role: FOLLOWER
I20260923 17:28:38.417016 13016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9899377b4a405ab48c9e6c5a19b37f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36395 } }
I20260923 17:28:38.417220 13016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:38.417325 13016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:38.417480 13016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:38.419947 13016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9899377b4a405ab48c9e6c5a19b37f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36395 } }
I20260923 17:28:38.420441 13016 leader_election.cc:302] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a9899377b4a405ab48c9e6c5a19b37f; no voters:
I20260923 17:28:38.421007 13016 leader_election.cc:288] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:38.421145 13021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:38.421839 13021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [term 1 LEADER]: Becoming Leader. State: Replica: 6a9899377b4a405ab48c9e6c5a19b37f, State: Running, Role: LEADER
I20260923 17:28:38.422394 13021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9899377b4a405ab48c9e6c5a19b37f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36395 } }
I20260923 17:28:38.423105 13016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:38.425107 13023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a9899377b4a405ab48c9e6c5a19b37f. Latest consensus state: current_term: 1 leader_uuid: "6a9899377b4a405ab48c9e6c5a19b37f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9899377b4a405ab48c9e6c5a19b37f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36395 } } }
I20260923 17:28:38.425238 13022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a9899377b4a405ab48c9e6c5a19b37f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a9899377b4a405ab48c9e6c5a19b37f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36395 } } }
I20260923 17:28:38.426002 13023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:38.426010 13022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:38.429486 13029 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:38.435170 13029 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:38.444521 13029 catalog_manager.cc:1347] Generated new cluster ID: 98dac90eea44418aa5cfe821c6513d5e
I20260923 17:28:38.444659 13029 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:38.458173 13029 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:38.458796 13029 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:38.468604 13029 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f: Generated new TSK 0
I20260923 17:28:38.469599 13029 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:38.484308 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36395
--builtin_ntp_servers=127.8.29.212:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 17:28:38.837520 13040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:38.837806 13040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:38.837894 13040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:38.845468 13040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:38.845584 13040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:38.845702 13040 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:38.855602 13040 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:38.856928 13040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:38.858719 13040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:38.871822 13045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:38.871874 13046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:38.871850 13048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:38.872480 13040 server_base.cc:1034] running on GCE node
I20260923 17:28:38.873114 13040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:38.874056 13040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:38.875267 13040 hybrid_clock.cc:648] HybridClock initialized: now 1790184518875214 us; error 40 us; skew 500 ppm
I20260923 17:28:38.875617 13040 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:28:38.878103 13040 webserver.cc:460] Webserver started at http://127.8.29.193:37571/ using document root <none> and password file <none>
I20260923 17:28:38.878916 13040 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:38.879015 13040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:38.879335 13040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:38.882536 13040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "f0a6eba958294a49b070f4883f41ae66"
format_stamp: "Formatted at 2026-09-23 17:28:38 on dist-test-slave-p7jt"
server_key: "7e29559bc22d55e00dceb5c72e399695"
server_key_iv: "c1ada1c71f2619e4aab58e589851231e"
server_key_version: "encryptionkey@0"
I20260923 17:28:38.883337 13040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "f0a6eba958294a49b070f4883f41ae66"
format_stamp: "Formatted at 2026-09-23 17:28:38 on dist-test-slave-p7jt"
server_key: "7e29559bc22d55e00dceb5c72e399695"
server_key_iv: "c1ada1c71f2619e4aab58e589851231e"
server_key_version: "encryptionkey@0"
I20260923 17:28:38.888870 13040 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:38.892688 13054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:38.894239 13040 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:38.894428 13040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "f0a6eba958294a49b070f4883f41ae66"
format_stamp: "Formatted at 2026-09-23 17:28:38 on dist-test-slave-p7jt"
server_key: "7e29559bc22d55e00dceb5c72e399695"
server_key_iv: "c1ada1c71f2619e4aab58e589851231e"
server_key_version: "encryptionkey@0"
I20260923 17:28:38.894810 13040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:38.901454 13040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:38.902428 13040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:38.902756 13040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:38.903789 13040 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:38.905421 13040 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:38.905537 13040 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:38.905658 13040 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:38.905738 13040 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:38.938396 13040 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:34477
I20260923 17:28:38.938434 13166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:34477 every 8 connection(s)
I20260923 17:28:38.939920 13040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:38.946584 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 13040
I20260923 17:28:38.946813 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:38.947502 8311 external_mini_cluster.cc:1428] Setting key 54037fb1e8077fca27e49fed0413bcbf
I20260923 17:28:38.948557 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:38.942610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:60115 (local address 127.8.29.254:36395)
0923 17:28:38.943064 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:38.943071 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:38.944348 (+ 1277us) server_negotiation.cc:407] Connection header received
0923 17:28:38.945995 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:38.946000 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:38.946057 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:38.946156 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:38.947803 (+ 1647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:38.947807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:38.947811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:38.947830 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:38.947885 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:38.947898 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:38.948006 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:38.948237 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 17:28:38.948304 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":210,"threads_started":1}
I20260923 17:28:38.949671 13170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:38.943272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:36395 (local address 127.8.29.193:60115)
0923 17:28:38.944205 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:38.944240 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:38.945718 (+ 1478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:38.946512 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:38.946526 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:38.946668 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:38.947422 (+ 754us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:38.947426 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:38.947483 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:38.947611 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:38.947999 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:38.948010 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:38.948018 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:28:38.948277 (+ 259us) client_negotiation.cc:240] Negotiation successful
0923 17:28:38.948543 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":280,"threads_started":1}
I20260923 17:28:38.950507 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36395
--builtin_ntp_servers=127.8.29.212:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 17:28:38.951182 13167 heartbeater.cc:347] Connected to a master server at 127.8.29.254:36395
I20260923 17:28:38.951529 13167 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:38.952235 13167 heartbeater.cc:511] Master 127.8.29.254:36395 requested a full tablet report, sending...
I20260923 17:28:38.954609 12981 ts_manager.cc:194] Registered new tserver with Master: f0a6eba958294a49b070f4883f41ae66 (127.8.29.193:34477)
W20260923 17:28:39.274178 13171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:39.274513 13171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:39.274642 13171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:39.282078 13171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:39.282191 13171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:39.282315 13171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:39.292516 13171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:39.293883 13171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:39.295766 13171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:39.309031 13177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:39.309062 13179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:39.309041 13176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:39.310995 13171 server_base.cc:1034] running on GCE node
I20260923 17:28:39.311647 13171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:39.312631 13171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:39.313874 13171 hybrid_clock.cc:648] HybridClock initialized: now 1790184519313809 us; error 52 us; skew 500 ppm
I20260923 17:28:39.314235 13171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:28:39.316884 13171 webserver.cc:460] Webserver started at http://127.8.29.194:33193/ using document root <none> and password file <none>
I20260923 17:28:39.317678 13171 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:39.317790 13171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:39.318166 13171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:39.321945 13171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "0fcf78d121b6448c91e1245fd870a5d9"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-p7jt"
server_key: "c1b2fefbde560c156a0b0b87f0d3f243"
server_key_iv: "c0d08979b7bbf562da8a8e533096d43d"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.322925 13171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "0fcf78d121b6448c91e1245fd870a5d9"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-p7jt"
server_key: "c1b2fefbde560c156a0b0b87f0d3f243"
server_key_iv: "c0d08979b7bbf562da8a8e533096d43d"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.329066 13171 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:28:39.333245 13185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.334869 13171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:28:39.335048 13171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "0fcf78d121b6448c91e1245fd870a5d9"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-p7jt"
server_key: "c1b2fefbde560c156a0b0b87f0d3f243"
server_key_iv: "c0d08979b7bbf562da8a8e533096d43d"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.335305 13171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:39.341346 13171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:39.342236 13171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:39.342540 13171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:39.343541 13171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:39.345046 13171 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:39.345151 13171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.345266 13171 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:39.345335 13171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.376482 13171 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:46243
I20260923 17:28:39.376570 13297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:46243 every 8 connection(s)
I20260923 17:28:39.377842 13171 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:39.380820 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 13171
I20260923 17:28:39.381007 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:39.381578 8311 external_mini_cluster.cc:1428] Setting key eb98d4d1f47c263f402121addaf9d869
I20260923 17:28:39.385154 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36395
--builtin_ntp_servers=127.8.29.212:40331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260923 17:28:39.387737 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.380599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:44227 (local address 127.8.29.254:36395)
0923 17:28:39.380764 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.380768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.382724 (+ 1956us) server_negotiation.cc:407] Connection header received
0923 17:28:39.385051 (+ 2327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.385056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.385138 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.385241 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.386936 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.386942 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.386948 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:39.386968 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.387008 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:39.387032 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.387150 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.387372 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.387436 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:28:39.389024 13300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.381310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:36395 (local address 127.8.29.194:44227)
0923 17:28:39.382565 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.382616 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.384703 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.385561 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.385575 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.385716 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:39.386560 (+ 844us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:39.386564 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:39.386622 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:39.386747 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:39.387138 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:39.387148 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:39.387156 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.387469 (+ 313us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.387770 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":379,"threads_started":1}
I20260923 17:28:39.390642 13298 heartbeater.cc:347] Connected to a master server at 127.8.29.254:36395
I20260923 17:28:39.390998 13298 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:39.391713 13298 heartbeater.cc:511] Master 127.8.29.254:36395 requested a full tablet report, sending...
I20260923 17:28:39.393371 12981 ts_manager.cc:194] Registered new tserver with Master: 0fcf78d121b6448c91e1245fd870a5d9 (127.8.29.194:46243)
W20260923 17:28:39.715423 13301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:39.715744 13301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:39.715881 13301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:39.723740 13301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:39.723937 13301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:39.724068 13301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:39.735047 13301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:40331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36395
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:39.736388 13301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:39.738306 13301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:39.752261 13306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:39.752293 13307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:39.752377 13309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:39.753008 13301 server_base.cc:1034] running on GCE node
I20260923 17:28:39.753582 13301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:39.754591 13301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:39.755792 13301 hybrid_clock.cc:648] HybridClock initialized: now 1790184519755743 us; error 30 us; skew 500 ppm
I20260923 17:28:39.756110 13301 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:28:39.758442 13301 webserver.cc:460] Webserver started at http://127.8.29.195:37391/ using document root <none> and password file <none>
I20260923 17:28:39.759159 13301 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:39.759246 13301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:39.759507 13301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:39.762527 13301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "3961d6e99efb4a2889539f27210d277e"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-p7jt"
server_key: "ecc18e82f8bf93bf34acdafd4c6cf91d"
server_key_iv: "e4095ce7461c52b0f91cca8d460698c2"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.763612 13301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "3961d6e99efb4a2889539f27210d277e"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-p7jt"
server_key: "ecc18e82f8bf93bf34acdafd4c6cf91d"
server_key_iv: "e4095ce7461c52b0f91cca8d460698c2"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.769644 13301 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:28:39.773697 13315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.775314 13301 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:39.775480 13301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "3961d6e99efb4a2889539f27210d277e"
format_stamp: "Formatted at 2026-09-23 17:28:39 on dist-test-slave-p7jt"
server_key: "ecc18e82f8bf93bf34acdafd4c6cf91d"
server_key_iv: "e4095ce7461c52b0f91cca8d460698c2"
server_key_version: "encryptionkey@0"
I20260923 17:28:39.775698 13301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:39.781723 13301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:39.782618 13301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:39.782847 13301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:39.783762 13301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:39.785132 13301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:39.785215 13301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.785297 13301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:39.785347 13301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:39.813685 13301 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:42047
I20260923 17:28:39.813736 13428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:42047 every 8 connection(s)
I20260923 17:28:39.815060 13301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:39.818792 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 13301
I20260923 17:28:39.818959 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:39.819384 8311 external_mini_cluster.cc:1428] Setting key c6eba4a8d295b9951e86f0d76646d337
I20260923 17:28:39.823944 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.817694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:42357 (local address 127.8.29.254:36395)
0923 17:28:39.817868 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.817873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.819563 (+ 1690us) server_negotiation.cc:407] Connection header received
0923 17:28:39.821564 (+ 2001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.821568 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.821640 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.821716 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.823213 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.823218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.823223 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:39.823243 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.823274 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:39.823285 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.823381 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.823566 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.823654 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:28:39.824911 13431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.818236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:36395 (local address 127.8.29.195:42357)
0923 17:28:39.819376 (+ 1140us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.819423 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.821290 (+ 1867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.821986 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.821998 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.822139 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:39.822873 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:39.822876 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:39.822941 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:39.823050 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:39.823366 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:39.823377 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:39.823384 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.823642 (+ 258us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.823935 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":368,"threads_started":1}
I20260923 17:28:39.826226 13429 heartbeater.cc:347] Connected to a master server at 127.8.29.254:36395
I20260923 17:28:39.826586 13429 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:39.827235 13429 heartbeater.cc:511] Master 127.8.29.254:36395 requested a full tablet report, sending...
I20260923 17:28:39.828649 12981 ts_manager.cc:194] Registered new tserver with Master: 3961d6e99efb4a2889539f27210d277e (127.8.29.195:42047)
I20260923 17:28:39.835347 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:28:39.842262 13169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.839874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40500 (local address 127.8.29.254:36395)
0923 17:28:39.840037 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.840042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.840529 (+ 487us) server_negotiation.cc:407] Connection header received
0923 17:28:39.840674 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.840678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.840737 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.840827 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.841677 (+ 850us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.841682 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.841685 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:39.841700 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.841733 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:39.841744 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.841880 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.842003 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.842052 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:28:39.846769 12981 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:39.853586 12981 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:39.872219 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.866615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55486 (local address 127.8.29.193:34477)
0923 17:28:39.867268 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.867278 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.867921 (+ 643us) server_negotiation.cc:407] Connection header received
0923 17:28:39.868047 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.868055 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.868275 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.868437 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.871102 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.871113 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.871406 (+ 293us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:39.871430 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.871499 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:39.871522 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.871651 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.871820 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.871929 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":382,"threads_started":1}
I20260923 17:28:39.872279 13441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.866408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:34477 (local address 127.0.0.1:55486)
0923 17:28:39.867752 (+ 1344us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.867796 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.867929 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.869752 (+ 1823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.869760 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.869865 (+ 105us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:39.870768 (+ 903us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:39.870771 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:39.870804 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:39.870887 (+ 83us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:39.871627 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:39.871638 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:39.871646 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.871816 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.871908 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1077,"thread_start_us":911,"threads_started":1}
I20260923 17:28:39.873144 13443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.868113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:42047 (local address 127.0.0.1:57418)
0923 17:28:39.868643 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.868659 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.868744 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.870849 (+ 2105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.870853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.870875 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:39.871426 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:39.871428 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:39.871443 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:39.871479 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:39.872660 (+ 1181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:39.872664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:39.872668 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.872769 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.872806 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":259,"threads_started":1}
I20260923 17:28:39.873255 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.868465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57418 (local address 127.8.29.195:42047)
0923 17:28:39.870316 (+ 1851us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.870323 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.870353 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:28:39.870422 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.870429 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.870666 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.871297 (+ 631us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.872444 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.872452 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.872465 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:39.872479 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.872533 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:39.872550 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.872718 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.872778 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.872873 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1685,"thread_start_us":2212,"threads_started":1}
I20260923 17:28:39.874300 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.868707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:46243 (local address 127.0.0.1:45576)
0923 17:28:39.869130 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:39.869218 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:39.869320 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:39.870579 (+ 1259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:39.870583 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:39.870605 (+ 22us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:39.871240 (+ 635us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:39.871243 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:39.871258 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:39.871299 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:39.873770 (+ 2471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:39.873774 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:39.873779 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:28:39.873900 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 17:28:39.873946 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":697,"threads_started":1}
I20260923 17:28:39.875176 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:39.869378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45576 (local address 127.8.29.194:46243)
0923 17:28:39.869961 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:39.869970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:39.870002 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:28:39.870071 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:39.870078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:39.870319 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:39.871019 (+ 700us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:39.873513 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:39.873535 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:39.873550 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:39.873577 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:39.873633 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:39.873655 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:39.873818 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:39.873881 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 17:28:39.873968 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":296,"threads_started":1}
I20260923 17:28:39.876765 13364 tablet_service.cc:1467] Processing CreateTablet for tablet d72eaf000c6c499ab3691bd94d85d346 (DEFAULT_TABLE table=test-table [id=fbe19c8d7f4f47f4b3df363050f9e203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:39.876787 13102 tablet_service.cc:1467] Processing CreateTablet for tablet d72eaf000c6c499ab3691bd94d85d346 (DEFAULT_TABLE table=test-table [id=fbe19c8d7f4f47f4b3df363050f9e203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:39.877954 13364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72eaf000c6c499ab3691bd94d85d346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:39.878111 13102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72eaf000c6c499ab3691bd94d85d346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:39.878571 13233 tablet_service.cc:1467] Processing CreateTablet for tablet d72eaf000c6c499ab3691bd94d85d346 (DEFAULT_TABLE table=test-table [id=fbe19c8d7f4f47f4b3df363050f9e203]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:39.879846 13233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d72eaf000c6c499ab3691bd94d85d346. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:39.889645 13447 tablet_bootstrap.cc:492] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: Bootstrap starting.
I20260923 17:28:39.892800 13448 tablet_bootstrap.cc:492] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66: Bootstrap starting.
I20260923 17:28:39.894148 13447 tablet_bootstrap.cc:654] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:39.895546 13447 log.cc:824] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:39.896503 13448 tablet_bootstrap.cc:654] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:39.897023 13449 tablet_bootstrap.cc:492] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: Bootstrap starting.
I20260923 17:28:39.897778 13448 log.cc:824] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:39.899139 13447 tablet_bootstrap.cc:492] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: No bootstrap required, opened a new log
I20260923 17:28:39.899428 13447 ts_tablet_manager.cc:1397] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 17:28:39.901261 13449 tablet_bootstrap.cc:654] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:39.902693 13449 log.cc:824] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:39.905761 13449 tablet_bootstrap.cc:492] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: No bootstrap required, opened a new log
I20260923 17:28:39.906051 13449 ts_tablet_manager.cc:1397] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 17:28:39.906124 13448 tablet_bootstrap.cc:492] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66: No bootstrap required, opened a new log
I20260923 17:28:39.906410 13448 ts_tablet_manager.cc:1397] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260923 17:28:39.908838 13447 raft_consensus.cc:348] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } }
I20260923 17:28:39.909319 13447 raft_consensus.cc:374] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:39.909413 13447 raft_consensus.cc:729] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3961d6e99efb4a2889539f27210d277e, State: Initialized, Role: FOLLOWER
I20260923 17:28:39.910089 13447 consensus_queue.cc:260] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } }
I20260923 17:28:39.911324 13429 heartbeater.cc:503] Master 127.8.29.254:36395 was elected leader, sending a full tablet report...
I20260923 17:28:39.911991 13447 ts_tablet_manager.cc:1428] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20260923 17:28:39.914721 13449 raft_consensus.cc:348] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } }
I20260923 17:28:39.915105 13449 raft_consensus.cc:374] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:39.915207 13449 raft_consensus.cc:729] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fcf78d121b6448c91e1245fd870a5d9, State: Initialized, Role: FOLLOWER
I20260923 17:28:39.915863 13449 consensus_queue.cc:260] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } }
I20260923 17:28:39.916013 13448 raft_consensus.cc:348] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } }
I20260923 17:28:39.916416 13448 raft_consensus.cc:374] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:39.916507 13448 raft_consensus.cc:729] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0a6eba958294a49b070f4883f41ae66, State: Initialized, Role: FOLLOWER
I20260923 17:28:39.917098 13448 consensus_queue.cc:260] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } }
I20260923 17:28:39.917672 13298 heartbeater.cc:503] Master 127.8.29.254:36395 was elected leader, sending a full tablet report...
I20260923 17:28:39.918473 13449 ts_tablet_manager.cc:1428] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: Time spent starting tablet: real 0.012s user 0.009s sys 0.004s
I20260923 17:28:39.919867 13167 heartbeater.cc:503] Master 127.8.29.254:36395 was elected leader, sending a full tablet report...
I20260923 17:28:39.920583 13448 ts_tablet_manager.cc:1428] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
W20260923 17:28:39.942245 13168 tablet.cc:2369] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:40.025447 13454 raft_consensus.cc:482] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:40.025774 13454 raft_consensus.cc:504] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } }
I20260923 17:28:40.027556 13454 leader_election.cc:288] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f0a6eba958294a49b070f4883f41ae66 (127.8.29.193:34477), 3961d6e99efb4a2889539f27210d277e (127.8.29.195:42047)
I20260923 17:28:40.031109 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.028105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:60781 (local address 127.8.29.193:34477)
0923 17:28:40.028342 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.028350 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.028823 (+ 473us) server_negotiation.cc:407] Connection header received
0923 17:28:40.029063 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.029071 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.029148 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.029273 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.030364 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.030370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.030374 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.030393 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.030437 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.030480 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.030609 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.030716 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.030785 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:28:40.031167 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.029115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:40567 (local address 127.8.29.195:42047)
0923 17:28:40.029293 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.029298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.029542 (+ 244us) server_negotiation.cc:407] Connection header received
0923 17:28:40.029674 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.029678 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.029729 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.029823 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.030603 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.030609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.030613 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.030631 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.030659 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.030671 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.030774 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.030852 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.030910 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:28:40.031320 13456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.027997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:34477 (local address 127.8.29.194:60781)
0923 17:28:40.028717 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.028733 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:40.028865 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:40.029450 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:40.029455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:40.029479 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:40.030169 (+ 690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:40.030172 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:40.030189 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:40.030229 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:40.030598 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:40.030604 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:40.030611 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:28:40.030773 (+ 162us) client_negotiation.cc:240] Negotiation successful
0923 17:28:40.030825 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":399,"threads_started":1}
I20260923 17:28:40.031423 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.028861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:42047 (local address 127.8.29.194:40567)
0923 17:28:40.029438 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.029454 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:40.029565 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:40.029856 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:40.029859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:40.029877 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:40.030403 (+ 526us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:28:40.030405 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:40.030416 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:28:40.030449 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:40.030755 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:40.030760 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:40.030765 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 17:28:40.030895 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 17:28:40.030942 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":256,"threads_started":1}
I20260923 17:28:40.032300 13384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72eaf000c6c499ab3691bd94d85d346" candidate_uuid: "0fcf78d121b6448c91e1245fd870a5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3961d6e99efb4a2889539f27210d277e" is_pre_election: true
I20260923 17:28:40.032766 13384 raft_consensus.cc:2454] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fcf78d121b6448c91e1245fd870a5d9 in term 0.
I20260923 17:28:40.033097 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72eaf000c6c499ab3691bd94d85d346" candidate_uuid: "0fcf78d121b6448c91e1245fd870a5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a6eba958294a49b070f4883f41ae66" is_pre_election: true
I20260923 17:28:40.033419 13186 leader_election.cc:302] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcf78d121b6448c91e1245fd870a5d9, 3961d6e99efb4a2889539f27210d277e; no voters:
I20260923 17:28:40.033660 13122 raft_consensus.cc:2454] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0fcf78d121b6448c91e1245fd870a5d9 in term 0.
I20260923 17:28:40.033931 13454 raft_consensus.cc:2781] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:40.034063 13454 raft_consensus.cc:482] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:40.034171 13454 raft_consensus.cc:3037] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:40.036379 13454 raft_consensus.cc:504] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } }
I20260923 17:28:40.036957 13454 leader_election.cc:288] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [CANDIDATE]: Term 1 election: Requested vote from peers f0a6eba958294a49b070f4883f41ae66 (127.8.29.193:34477), 3961d6e99efb4a2889539f27210d277e (127.8.29.195:42047)
I20260923 17:28:40.037400 13122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72eaf000c6c499ab3691bd94d85d346" candidate_uuid: "0fcf78d121b6448c91e1245fd870a5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0a6eba958294a49b070f4883f41ae66"
I20260923 17:28:40.037451 13384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d72eaf000c6c499ab3691bd94d85d346" candidate_uuid: "0fcf78d121b6448c91e1245fd870a5d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3961d6e99efb4a2889539f27210d277e"
I20260923 17:28:40.037626 13122 raft_consensus.cc:3037] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:40.037640 13384 raft_consensus.cc:3037] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:40.039888 13122 raft_consensus.cc:2454] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fcf78d121b6448c91e1245fd870a5d9 in term 1.
I20260923 17:28:40.040153 13384 raft_consensus.cc:2454] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0fcf78d121b6448c91e1245fd870a5d9 in term 1.
I20260923 17:28:40.040302 13186 leader_election.cc:302] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0fcf78d121b6448c91e1245fd870a5d9, f0a6eba958294a49b070f4883f41ae66; no voters:
I20260923 17:28:40.040591 13454 raft_consensus.cc:2781] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:40.041026 13454 raft_consensus.cc:686] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 1 LEADER]: Becoming Leader. State: Replica: 0fcf78d121b6448c91e1245fd870a5d9, State: Running, Role: LEADER
I20260923 17:28:40.041476 13454 consensus_queue.cc:237] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } }
I20260923 17:28:40.046016 12980 catalog_manager.cc:5495] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0fcf78d121b6448c91e1245fd870a5d9 (127.8.29.194). New cstate: current_term: 1 leader_uuid: "0fcf78d121b6448c91e1245fd870a5d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0fcf78d121b6448c91e1245fd870a5d9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 46243 } health_report { overall_health: HEALTHY } } }
W20260923 17:28:40.066737 13430 tablet.cc:2369] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:40.070091 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.068270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.8.29.194:46243)
0923 17:28:40.068456 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.068461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.068535 (+ 74us) server_negotiation.cc:407] Connection header received
0923 17:28:40.068717 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.068720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.068807 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.068888 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.069601 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.069607 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.069611 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.069625 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.069654 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.069666 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.069763 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.069815 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.069859 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:28:40.078897 13122 raft_consensus.cc:1261] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [term 1 FOLLOWER]: Refusing update from remote peer 0fcf78d121b6448c91e1245fd870a5d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:40.078922 13384 raft_consensus.cc:1261] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [term 1 FOLLOWER]: Refusing update from remote peer 0fcf78d121b6448c91e1245fd870a5d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:40.079937 13458 consensus_queue.cc:1035] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0a6eba958294a49b070f4883f41ae66" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 34477 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:40.080333 13454 consensus_queue.cc:1035] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3961d6e99efb4a2889539f27210d277e" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 42047 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:40.107036 12980 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:40.107323 12980 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:40.110769 12980 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a9899377b4a405ab48c9e6c5a19b37f: Sending DeleteTablet for 3 replicas of tablet d72eaf000c6c499ab3691bd94d85d346
I20260923 17:28:40.111873 13102 tablet_service.cc:1514] Processing DeleteTablet for tablet d72eaf000c6c499ab3691bd94d85d346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:40 UTC) from {username='slave'} at 127.0.0.1:55486
I20260923 17:28:40.112002 13364 tablet_service.cc:1514] Processing DeleteTablet for tablet d72eaf000c6c499ab3691bd94d85d346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:40 UTC) from {username='slave'} at 127.0.0.1:57418
I20260923 17:28:40.113168 13233 tablet_service.cc:1514] Processing DeleteTablet for tablet d72eaf000c6c499ab3691bd94d85d346 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:40 UTC) from {username='slave'} at 127.0.0.1:45576
I20260923 17:28:40.114038 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 13040
I20260923 17:28:40.123050 13471 tablet_replica.cc:330] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66: stopping tablet replica
I20260923 17:28:40.124110 13473 tablet_replica.cc:330] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: stopping tablet replica
I20260923 17:28:40.124636 13473 raft_consensus.cc:2229] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:40.124712 13471 raft_consensus.cc:2229] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:40.125438 13473 raft_consensus.cc:2258] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:40.126066 13442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.117758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.8.29.193:34477)
0923 17:28:40.117986 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.117993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.123392 (+ 5399us) server_negotiation.cc:407] Connection header received
0923 17:28:40.123614 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.123620 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.123740 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.123880 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.125239 (+ 1359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.125246 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.125251 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.125276 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.125320 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.125336 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.125439 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.125696 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.125769 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260923 17:28:40.126217 13472 tablet_replica.cc:330] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: stopping tablet replica
I20260923 17:28:40.126866 13471 raft_consensus.cc:2258] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:40.127146 13162 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:40.128072 13473 ts_tablet_manager.cc:1905] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:40.128100 13472 raft_consensus.cc:2229] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:40.128727 13472 raft_consensus.cc:2258] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:40.131832 13472 ts_tablet_manager.cc:1905] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:40.135793 13473 ts_tablet_manager.cc:1918] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:40.135995 13473 log.cc:1196] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/d72eaf000c6c499ab3691bd94d85d346
I20260923 17:28:40.136545 13473 ts_tablet_manager.cc:1939] T d72eaf000c6c499ab3691bd94d85d346 P 0fcf78d121b6448c91e1245fd870a5d9: Deleting consensus metadata
I20260923 17:28:40.136678 13472 ts_tablet_manager.cc:1918] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:40.136837 13472 log.cc:1196] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/d72eaf000c6c499ab3691bd94d85d346
I20260923 17:28:40.137223 13472 ts_tablet_manager.cc:1939] T d72eaf000c6c499ab3691bd94d85d346 P 3961d6e99efb4a2889539f27210d277e: Deleting consensus metadata
I20260923 17:28:40.138624 12965 catalog_manager.cc:4841] TS 0fcf78d121b6448c91e1245fd870a5d9 (127.8.29.194:46243): tablet d72eaf000c6c499ab3691bd94d85d346 (table test-table [id=fbe19c8d7f4f47f4b3df363050f9e203]) successfully deleted
I20260923 17:28:40.139513 12965 catalog_manager.cc:4841] TS 3961d6e99efb4a2889539f27210d277e (127.8.29.195:42047): tablet d72eaf000c6c499ab3691bd94d85d346 (table test-table [id=fbe19c8d7f4f47f4b3df363050f9e203]) successfully deleted
I20260923 17:28:40.224596 13471 ts_tablet_manager.cc:1905] T d72eaf000c6c499ab3691bd94d85d346 P f0a6eba958294a49b070f4883f41ae66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:40.229067 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 13040
I20260923 17:28:40.249819 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 13171
W20260923 17:28:40.249905 12965 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.29.193:34477 (error 108)
I20260923 17:28:40.252355 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.251862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:34477 (local address 127.0.0.1:55506)
0923 17:28:40.252041 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.252184 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:34477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260923 17:28:40.252624 12965 catalog_manager.cc:4584] TS f0a6eba958294a49b070f4883f41ae66 (127.8.29.193:34477): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d72eaf000c6c499ab3691bd94d85d346: Network error: Client connection negotiation failed: client connection to 127.8.29.193:34477: connect: Connection refused (error 111)
I20260923 17:28:40.253170 13446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.250701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.8.29.194:46243)
0923 17:28:40.250897 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.250903 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.251213 (+ 310us) server_negotiation.cc:407] Connection header received
0923 17:28:40.251312 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.251317 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.251387 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.251480 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.252400 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.252406 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.252410 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.252429 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.252469 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.252486 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.252612 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.252809 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.252892 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:28:40.253820 13293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:40.306694 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.306031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:34477 (local address 127.0.0.1:55520)
0923 17:28:40.306306 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.306434 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:34477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260923 17:28:40.307924 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.307497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:34477 (local address 127.0.0.1:55528)
0923 17:28:40.307676 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.307773 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:34477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:28:40.363555 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 13171
I20260923 17:28:40.368554 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.368130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:34477 (local address 127.0.0.1:55540)
0923 17:28:40.368303 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.368393 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:34477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 17:28:40.369546 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.369217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:34477 (local address 127.0.0.1:55556)
0923 17:28:40.369362 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.369430 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:34477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 17:28:40.377830 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 13301
I20260923 17:28:40.380393 13444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.378549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57422 (local address 127.8.29.195:42047)
0923 17:28:40.378727 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:40.378732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:40.378750 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:40.378883 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:40.378887 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:40.378959 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:40.379039 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:40.379843 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:40.379846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:40.379849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:40.379860 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:40.379903 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:40.379915 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:40.380027 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:40.380080 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 17:28:40.380124 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:28:40.380965 13424 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:40.450012 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.449426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:34477 (local address 127.0.0.1:55570)
0923 17:28:40.449714 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.449836 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:34477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260923 17:28:40.451224 13445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:40.450765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:34477 (local address 127.0.0.1:55574)
0923 17:28:40.450987 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:40.451072 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:34477: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260923 17:28:40.471551 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 13301
I20260923 17:28:40.484298 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 12950
I20260923 17:28:40.485468 13011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:40.591724 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 12950
2026-09-23T17:28:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2674 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 17:28:40.628585 8311 mini_oidc.cc:152] Starting JWKS server
I20260923 17:28:40.631140 8311 webserver.cc:460] Webserver started at https://127.0.0.1:38705/ using document root <none> and password file <none>
I20260923 17:28:40.631402 8311 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:28:40.631982 8311 webserver.cc:460] Webserver started at http://127.0.0.1:41247/ using document root <none> and password file <none>
2026-09-23T17:28:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:40Z Disabled control of system clock
I20260923 17:28:40.657141 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:32897
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:45283
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:32897
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38705/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/testchainca.cert with env {}
W20260923 17:28:40.982354 13485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:40.982702 13485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:40.982829 13485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:40.990162 13485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:40.990306 13485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:40.990376 13485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:40.990475 13485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:40.990685 13485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:28:40.990804 13485 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:28:40.990862 13485 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38705/jwks.json
I20260923 17:28:41.001674 13485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45283
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:32897
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:32897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38705/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:41.003163 13485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:41.004528 13485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:41.016657 13490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:41.016688 13493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:41.016733 13491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:41.017776 13485 server_base.cc:1034] running on GCE node
I20260923 17:28:41.018688 13485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:41.020205 13485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:41.021446 13485 hybrid_clock.cc:648] HybridClock initialized: now 1790184521021389 us; error 39 us; skew 500 ppm
I20260923 17:28:41.021872 13485 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:28:41.024371 13485 webserver.cc:460] Webserver started at http://127.8.29.254:37821/ using document root <none> and password file <none>
I20260923 17:28:41.025087 13485 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:41.025183 13485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:41.025523 13485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:41.028492 13485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "5bac9ad7c8a44425b9e2a386deedface"
format_stamp: "Formatted at 2026-09-23 17:28:41 on dist-test-slave-p7jt"
server_key: "74dba48b0075ceec99c2c1ca52467b40"
server_key_iv: "b92b04b4b914528b5ade3cc43ba08223"
server_key_version: "encryptionkey@0"
I20260923 17:28:41.029271 13485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "5bac9ad7c8a44425b9e2a386deedface"
format_stamp: "Formatted at 2026-09-23 17:28:41 on dist-test-slave-p7jt"
server_key: "74dba48b0075ceec99c2c1ca52467b40"
server_key_iv: "b92b04b4b914528b5ade3cc43ba08223"
server_key_version: "encryptionkey@0"
I20260923 17:28:41.034857 13485 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:28:41.038339 13499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:41.039815 13485 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:41.040045 13485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "5bac9ad7c8a44425b9e2a386deedface"
format_stamp: "Formatted at 2026-09-23 17:28:41 on dist-test-slave-p7jt"
server_key: "74dba48b0075ceec99c2c1ca52467b40"
server_key_iv: "b92b04b4b914528b5ade3cc43ba08223"
server_key_version: "encryptionkey@0"
I20260923 17:28:41.040289 13485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:41.058589 13485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:41.059497 13485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:41.059803 13485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:41.077287 13485 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:32897
I20260923 17:28:41.077308 13550 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:32897 every 8 connection(s)
I20260923 17:28:41.078727 13485 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:41.082507 13551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:41.085247 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 13485
I20260923 17:28:41.085444 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:41.086234 8311 external_mini_cluster.cc:1428] Setting key 5ef18ea12a5fe4c6b3e8ebe0786c516a
I20260923 17:28:41.094543 13551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface: Bootstrap starting.
I20260923 17:28:41.097800 13551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:41.098979 13551 log.cc:824] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:41.100099 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.088374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52680 (local address 127.8.29.254:32897)
0923 17:28:41.089370 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:41.089382 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:41.089424 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:28:41.091159 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:41.091188 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:41.091705 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:41.092111 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:41.092734 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.093980 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.095389 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.096093 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.096268 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.097475 (+ 1207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:41.097483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:41.097498 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:41.098103 (+ 605us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:41.098210 (+ 107us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:41.098233 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:41.098339 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:41.098593 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 17:28:41.098915 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":308,"threads_started":1}
I20260923 17:28:41.102113 13551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface: No bootstrap required, opened a new log
I20260923 17:28:41.104169 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:41.110213 13551 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac9ad7c8a44425b9e2a386deedface" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32897 } }
I20260923 17:28:41.110667 13551 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:41.110767 13551 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bac9ad7c8a44425b9e2a386deedface, State: Initialized, Role: FOLLOWER
I20260923 17:28:41.111426 13551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac9ad7c8a44425b9e2a386deedface" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32897 } }
I20260923 17:28:41.111722 13551 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:41.111866 13551 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:41.112036 13551 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:41.114432 13551 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac9ad7c8a44425b9e2a386deedface" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32897 } }
I20260923 17:28:41.114985 13551 leader_election.cc:302] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bac9ad7c8a44425b9e2a386deedface; no voters:
I20260923 17:28:41.115541 13551 leader_election.cc:288] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:41.115705 13556 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:41.116392 13556 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [term 1 LEADER]: Becoming Leader. State: Replica: 5bac9ad7c8a44425b9e2a386deedface, State: Running, Role: LEADER
I20260923 17:28:41.116890 13556 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac9ad7c8a44425b9e2a386deedface" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32897 } }
I20260923 17:28:41.117556 13551 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:41.119566 13557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bac9ad7c8a44425b9e2a386deedface" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac9ad7c8a44425b9e2a386deedface" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32897 } } }
I20260923 17:28:41.119958 13557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:41.119995 13558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bac9ad7c8a44425b9e2a386deedface. Latest consensus state: current_term: 1 leader_uuid: "5bac9ad7c8a44425b9e2a386deedface" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bac9ad7c8a44425b9e2a386deedface" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32897 } } }
I20260923 17:28:41.120343 13558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:41.123824 13561 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:41.129015 13561 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:41.137555 13561 catalog_manager.cc:1347] Generated new cluster ID: 1e48a0fd88804ae28bc62ffb5384c1bc
I20260923 17:28:41.137689 13561 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:41.158993 13561 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:41.160267 13561 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:41.166725 13561 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bac9ad7c8a44425b9e2a386deedface: Generated new TSK 0
I20260923 17:28:41.167539 13561 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 17:28:41.219888 13581 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 17:28:41.220090 13581 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:28:41.220858 13554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.191757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52686 (local address 127.8.29.254:32897)
0923 17:28:41.191981 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:41.191989 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:41.192526 (+ 537us) server_negotiation.cc:407] Connection header received
0923 17:28:41.192880 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:41.192884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:41.192957 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:41.193085 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:28:41.193727 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.194497 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.196652 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.197096 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.197207 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.197812 (+ 605us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:28:41.220308 (+ 22496us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:28:41.220627 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:28:41.220973 13554 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52686: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 17:28:41.222954 13580 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:41.191699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32897 (local address 127.0.0.1:52686)
0923 17:28:41.192412 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:41.192437 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:41.192633 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:41.193156 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:41.193162 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:41.193176 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:28:41.193558 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:41.193571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.194652 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.194657 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:41.196475 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:41.196490 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.197272 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.197279 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:41.197468 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.197555 (+ 87us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:28:41.220604 (+ 23049us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 17:28:41.221004 (+ 400us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.29.254:32897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":326,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.29.254:32897: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38705/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 17:28:41.225052 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 13485
I20260923 17:28:41.226219 13546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:41.320353 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 13485
2026-09-23T17:28:41Z chronyd exiting
I20260923 17:28:41.367967 8311 test_util.cc:175] -----------------------------------------------
I20260923 17:28:41.368093 8311 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790184478467572-8311-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (743 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 17:28:41.371781 8311 mini_oidc.cc:152] Starting JWKS server
I20260923 17:28:41.373587 8311 webserver.cc:460] Webserver started at https://127.0.0.1:45439/ using document root <none> and password file <none>
I20260923 17:28:41.373805 8311 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:28:41.374405 8311 webserver.cc:460] Webserver started at http://127.0.0.1:37973/ using document root <none> and password file <none>
2026-09-23T17:28:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:41Z Disabled control of system clock
I20260923 17:28:41.399062 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:35801
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:34177
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:35801
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45439/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/testchainca.cert with env {}
W20260923 17:28:41.715839 13588 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:41.716174 13588 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:41.716305 13588 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:41.723634 13588 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:41.723775 13588 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:41.723838 13588 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:41.723917 13588 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:41.723977 13588 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:28:41.724056 13588 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:28:41.724115 13588 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:45439/jwks.json
I20260923 17:28:41.735385 13588 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:34177
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:35801
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:35801
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:45439/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:41.736734 13588 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:41.738055 13588 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:41.749938 13596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:41.749979 13593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:41.750814 13594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:41.752583 13588 server_base.cc:1034] running on GCE node
I20260923 17:28:41.753777 13588 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:41.755802 13588 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:41.757167 13588 hybrid_clock.cc:648] HybridClock initialized: now 1790184521757093 us; error 61 us; skew 500 ppm
I20260923 17:28:41.757695 13588 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:28:41.760895 13588 webserver.cc:460] Webserver started at http://127.8.29.254:41979/ using document root <none> and password file <none>
I20260923 17:28:41.762012 13588 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:41.762154 13588 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:41.762579 13588 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:41.766677 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "651f0d9adfa74adbadbcf7da7553d419"
format_stamp: "Formatted at 2026-09-23 17:28:41 on dist-test-slave-p7jt"
server_key: "f04322dc1043f2ab92e8e79768849839"
server_key_iv: "73098f551ed0babf4f2e6b926d935c36"
server_key_version: "encryptionkey@0"
I20260923 17:28:41.767486 13588 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "651f0d9adfa74adbadbcf7da7553d419"
format_stamp: "Formatted at 2026-09-23 17:28:41 on dist-test-slave-p7jt"
server_key: "f04322dc1043f2ab92e8e79768849839"
server_key_iv: "73098f551ed0babf4f2e6b926d935c36"
server_key_version: "encryptionkey@0"
I20260923 17:28:41.774550 13588 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260923 17:28:41.779348 13602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:41.781023 13588 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:41.781229 13588 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "651f0d9adfa74adbadbcf7da7553d419"
format_stamp: "Formatted at 2026-09-23 17:28:41 on dist-test-slave-p7jt"
server_key: "f04322dc1043f2ab92e8e79768849839"
server_key_iv: "73098f551ed0babf4f2e6b926d935c36"
server_key_version: "encryptionkey@0"
I20260923 17:28:41.781450 13588 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:41.794669 13588 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:41.795511 13588 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:41.795787 13588 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:41.812568 13588 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:35801
I20260923 17:28:41.812573 13653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:35801 every 8 connection(s)
I20260923 17:28:41.813944 13588 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:41.817062 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 13588
I20260923 17:28:41.817257 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:41.817819 13654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:41.817955 8311 external_mini_cluster.cc:1428] Setting key da6908f63a69d881b8c2cdbd42aeb213
I20260923 17:28:41.830425 13654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419: Bootstrap starting.
I20260923 17:28:41.830996 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.819799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.8.29.254:35801)
0923 17:28:41.820841 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:41.820853 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:41.820897 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:28:41.822629 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:41.822651 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:41.823160 (+ 509us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:41.823537 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:41.824134 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.825393 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.826668 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.827350 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.827524 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.828685 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:41.828693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:41.828707 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:41.829246 (+ 539us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:41.829323 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:41.829344 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:41.829441 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:41.829658 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 17:28:41.829958 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":372,"threads_started":1}
I20260923 17:28:41.834031 13654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:41.835175 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:41.835213 13654 log.cc:824] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:41.837944 13654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419: No bootstrap required, opened a new log
I20260923 17:28:41.845238 13654 raft_consensus.cc:348] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f0d9adfa74adbadbcf7da7553d419" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35801 } }
I20260923 17:28:41.845636 13654 raft_consensus.cc:374] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:41.845736 13654 raft_consensus.cc:729] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651f0d9adfa74adbadbcf7da7553d419, State: Initialized, Role: FOLLOWER
I20260923 17:28:41.846303 13654 consensus_queue.cc:260] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f0d9adfa74adbadbcf7da7553d419" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35801 } }
I20260923 17:28:41.846553 13654 raft_consensus.cc:388] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:41.846688 13654 raft_consensus.cc:482] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:41.846835 13654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:41.848941 13654 raft_consensus.cc:504] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f0d9adfa74adbadbcf7da7553d419" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35801 } }
I20260923 17:28:41.849459 13654 leader_election.cc:302] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 651f0d9adfa74adbadbcf7da7553d419; no voters:
I20260923 17:28:41.850035 13654 leader_election.cc:288] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:41.850219 13659 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:41.851013 13659 raft_consensus.cc:686] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [term 1 LEADER]: Becoming Leader. State: Replica: 651f0d9adfa74adbadbcf7da7553d419, State: Running, Role: LEADER
I20260923 17:28:41.851653 13659 consensus_queue.cc:237] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f0d9adfa74adbadbcf7da7553d419" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35801 } }
I20260923 17:28:41.852325 13654 sys_catalog.cc:564] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:41.854822 13660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "651f0d9adfa74adbadbcf7da7553d419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f0d9adfa74adbadbcf7da7553d419" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35801 } } }
I20260923 17:28:41.854789 13661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 651f0d9adfa74adbadbcf7da7553d419. Latest consensus state: current_term: 1 leader_uuid: "651f0d9adfa74adbadbcf7da7553d419" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "651f0d9adfa74adbadbcf7da7553d419" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35801 } } }
I20260923 17:28:41.855273 13661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:41.855273 13660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:41.856206 13668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:41.862080 13668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:41.871294 13668 catalog_manager.cc:1347] Generated new cluster ID: b7563ad995b54ff094bc3168082626f4
I20260923 17:28:41.871447 13668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:41.880561 13668 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:41.882421 13668 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:41.892331 13668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 651f0d9adfa74adbadbcf7da7553d419: Generated new TSK 0
I20260923 17:28:41.893268 13668 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 17:28:41.932017 13684 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 17:28:41.932144 13684 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:28:41.933483 13657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:41.912908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54408 (local address 127.8.29.254:35801)
0923 17:28:41.913130 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:41.913138 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:41.913407 (+ 269us) server_negotiation.cc:407] Connection header received
0923 17:28:41.913636 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:41.913642 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:41.913739 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:41.913928 (+ 189us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:28:41.914327 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.915497 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.916967 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.917653 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.917784 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.918327 (+ 543us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:28:41.932741 (+ 14414us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:28:41.933159 (+ 418us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":85}
W20260923 17:28:41.933614 13657 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54408: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 17:28:41.933728 13683 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:41.912713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35801 (local address 127.0.0.1:54408)
0923 17:28:41.913298 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:41.913317 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:41.913454 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:41.913922 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:41.913926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:41.913935 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:28:41.914173 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:41.914180 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.915683 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.915688 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:41.916810 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:41.916823 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:41.917839 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:41.917844 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:41.917986 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:41.918023 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:28:41.933013 (+ 14990us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 17:28:41.933374 (+ 361us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.29.254:35801: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:45439/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":309,"threads_started":1}
I20260923 17:28:41.934701 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 13588
I20260923 17:28:41.935909 13649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:42.027284 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 13588
2026-09-23T17:28:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (707 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 17:28:42.079718 8311 mini_oidc.cc:152] Starting JWKS server
I20260923 17:28:42.081657 8311 webserver.cc:460] Webserver started at https://127.0.0.1:38167/ using document root <none> and password file <none>
I20260923 17:28:42.081877 8311 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 17:28:42.082609 8311 webserver.cc:460] Webserver started at http://127.0.0.1:38079/ using document root <none> and password file <none>
2026-09-23T17:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:42Z Disabled control of system clock
I20260923 17:28:42.107487 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:33093
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:37581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:33093
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38167/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:28:42.425933 13691 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:42.426258 13691 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:42.426345 13691 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:42.433722 13691 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:42.433858 13691 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:42.433913 13691 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:42.433996 13691 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:42.434062 13691 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:28:42.434128 13691 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:28:42.434168 13691 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38167/jwks.json
I20260923 17:28:42.443887 13691 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:37581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:33093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:33093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38167/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:42.445222 13691 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:42.447038 13691 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:42.458999 13696 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:42.459084 13697 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:42.459100 13699 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:42.459856 13691 server_base.cc:1034] running on GCE node
I20260923 17:28:42.460675 13691 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:42.462114 13691 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:42.463351 13691 hybrid_clock.cc:648] HybridClock initialized: now 1790184522463298 us; error 41 us; skew 500 ppm
I20260923 17:28:42.463719 13691 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:28:42.473091 13691 webserver.cc:460] Webserver started at http://127.8.29.254:34703/ using document root <none> and password file <none>
I20260923 17:28:42.473866 13691 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:42.473978 13691 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:42.474323 13691 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:42.477425 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "b4d6d91700924e458f29299d8070c755"
format_stamp: "Formatted at 2026-09-23 17:28:42 on dist-test-slave-p7jt"
server_key: "8df8e91d856bedab1c953baad781454d"
server_key_iv: "9dc15df1553cc439db1fe1d2f3cb591a"
server_key_version: "encryptionkey@0"
I20260923 17:28:42.478247 13691 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "b4d6d91700924e458f29299d8070c755"
format_stamp: "Formatted at 2026-09-23 17:28:42 on dist-test-slave-p7jt"
server_key: "8df8e91d856bedab1c953baad781454d"
server_key_iv: "9dc15df1553cc439db1fe1d2f3cb591a"
server_key_version: "encryptionkey@0"
I20260923 17:28:42.483479 13691 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:28:42.487196 13705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:42.488646 13691 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:42.488817 13691 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "b4d6d91700924e458f29299d8070c755"
format_stamp: "Formatted at 2026-09-23 17:28:42 on dist-test-slave-p7jt"
server_key: "8df8e91d856bedab1c953baad781454d"
server_key_iv: "9dc15df1553cc439db1fe1d2f3cb591a"
server_key_version: "encryptionkey@0"
I20260923 17:28:42.489029 13691 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:42.502422 13691 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:42.503348 13691 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:42.503639 13691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:42.520919 13691 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:33093
I20260923 17:28:42.520951 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:33093 every 8 connection(s)
I20260923 17:28:42.522634 13691 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:42.525904 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 13691
I20260923 17:28:42.526103 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:42.526922 8311 external_mini_cluster.cc:1428] Setting key a7d2c337af41c78136bf1180fdab6f67
I20260923 17:28:42.527349 13758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:42.540927 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:42.528990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44020 (local address 127.8.29.254:33093)
0923 17:28:42.529875 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:42.529886 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:42.529931 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:28:42.531534 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:42.531556 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:42.532059 (+ 503us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:42.532413 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:42.533062 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.534453 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.536314 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.537031 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.537167 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:42.538398 (+ 1231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:42.538406 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:42.538423 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:42.539083 (+ 660us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:42.539171 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:42.539197 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:42.539307 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:42.539525 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 17:28:42.539877 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":278,"threads_started":1}
I20260923 17:28:42.541678 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755: Bootstrap starting.
I20260923 17:28:42.544643 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:42.544953 13758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:42.545897 13758 log.cc:824] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:42.548768 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755: No bootstrap required, opened a new log
I20260923 17:28:42.555913 13758 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d6d91700924e458f29299d8070c755" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33093 } }
I20260923 17:28:42.556295 13758 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:42.556387 13758 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4d6d91700924e458f29299d8070c755, State: Initialized, Role: FOLLOWER
I20260923 17:28:42.557024 13758 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d6d91700924e458f29299d8070c755" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33093 } }
I20260923 17:28:42.557287 13758 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:42.557412 13758 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:42.557569 13758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:42.559700 13758 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d6d91700924e458f29299d8070c755" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33093 } }
I20260923 17:28:42.560178 13758 leader_election.cc:302] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4d6d91700924e458f29299d8070c755; no voters:
I20260923 17:28:42.560781 13758 leader_election.cc:288] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:42.560968 13763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:42.561872 13763 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [term 1 LEADER]: Becoming Leader. State: Replica: b4d6d91700924e458f29299d8070c755, State: Running, Role: LEADER
I20260923 17:28:42.562446 13763 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d6d91700924e458f29299d8070c755" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33093 } }
I20260923 17:28:42.563015 13758 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:42.565493 13765 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4d6d91700924e458f29299d8070c755. Latest consensus state: current_term: 1 leader_uuid: "b4d6d91700924e458f29299d8070c755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d6d91700924e458f29299d8070c755" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33093 } } }
I20260923 17:28:42.565549 13764 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4d6d91700924e458f29299d8070c755" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4d6d91700924e458f29299d8070c755" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33093 } } }
I20260923 17:28:42.565933 13764 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:42.565935 13765 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:42.567806 13770 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:42.572825 13770 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:42.582170 13770 catalog_manager.cc:1347] Generated new cluster ID: d0979cad268a497aa281ef4cd889c9eb
I20260923 17:28:42.582297 13770 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:42.598717 13770 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:42.600730 13770 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:42.612210 13770 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4d6d91700924e458f29299d8070c755: Generated new TSK 0
I20260923 17:28:42.613097 13770 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:42.644550 13761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:42.635956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44036 (local address 127.8.29.254:33093)
0923 17:28:42.636171 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:42.636177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:42.636495 (+ 318us) server_negotiation.cc:407] Connection header received
0923 17:28:42.636729 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:42.636735 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:42.636838 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:42.637014 (+ 176us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:28:42.637400 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.638620 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.640062 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.640773 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.640903 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:42.641509 (+ 606us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:28:42.643902 (+ 2393us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:28:42.644240 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44036: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":90}
W20260923 17:28:42.644688 13761 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44036: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260923 17:28:42.646485 13787 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:42.635800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33093 (local address 127.0.0.1:44036)
0923 17:28:42.636392 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:42.636407 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:42.636537 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:42.637011 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:42.637014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:42.637023 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:28:42.637259 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:42.637266 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.638778 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.638783 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:42.639912 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:42.639925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:42.641022 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:42.641029 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:42.641176 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:42.641216 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:28:42.644172 (+ 2956us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 17:28:42.646042 (+ 1870us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.29.254:33093: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38167/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":1519,"thread_start_us":319,"threads_started":1}
I20260923 17:28:42.647796 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 13691
I20260923 17:28:42.649195 13753 generic_service.cc:196] Checking for leaks (request via RPC)
E20260923 17:28:42.656311 13788 webserver.cc:533] Webserver: sslize failed:
E20260923 17:28:42.656422 13788 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 17:28:42.742197 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 13691
2026-09-23T17:28:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (714 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T17:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:42Z Disabled control of system clock
I20260923 17:28:42.818667 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:35781
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:41385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:35781
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/testchainca.cert with env {}
W20260923 17:28:43.142112 13793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:43.142436 13793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:43.142575 13793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:43.149906 13793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:43.150012 13793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:43.150048 13793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:43.150094 13793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:43.150141 13793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 17:28:43.150198 13793 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 17:28:43.150236 13793 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 17:28:43.160239 13793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:41385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:35781
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:35781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:43.161504 13793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:43.162810 13793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:43.174101 13798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:43.174124 13799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:43.174189 13801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:43.175259 13793 server_base.cc:1034] running on GCE node
I20260923 17:28:43.176065 13793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:43.177526 13793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:43.178732 13793 hybrid_clock.cc:648] HybridClock initialized: now 1790184523178689 us; error 38 us; skew 500 ppm
I20260923 17:28:43.179080 13793 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:28:43.181286 13793 webserver.cc:460] Webserver started at http://127.8.29.254:37657/ using document root <none> and password file <none>
I20260923 17:28:43.181993 13793 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:43.182094 13793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:43.182389 13793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:43.185209 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "5f2c594a576f428b974878a194c4a49e"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-p7jt"
server_key: "a1b6402e7d735e513f75277309625ede"
server_key_iv: "dd8669466b51e91585adc5e82c79e093"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.185995 13793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "5f2c594a576f428b974878a194c4a49e"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-p7jt"
server_key: "a1b6402e7d735e513f75277309625ede"
server_key_iv: "dd8669466b51e91585adc5e82c79e093"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.191030 13793 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:28:43.194435 13807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:43.195876 13793 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:43.196039 13793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "5f2c594a576f428b974878a194c4a49e"
format_stamp: "Formatted at 2026-09-23 17:28:43 on dist-test-slave-p7jt"
server_key: "a1b6402e7d735e513f75277309625ede"
server_key_iv: "dd8669466b51e91585adc5e82c79e093"
server_key_version: "encryptionkey@0"
I20260923 17:28:43.196246 13793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:43.241581 13793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:43.242542 13793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:43.242810 13793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:43.259434 13793 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:35781
I20260923 17:28:43.259454 13858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:35781 every 8 connection(s)
I20260923 17:28:43.260813 13793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:43.264498 13859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:43.268460 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 13793
I20260923 17:28:43.268673 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:43.269443 8311 external_mini_cluster.cc:1428] Setting key 8b9c6a045759747b155f0d59234874f4
I20260923 17:28:43.276382 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e: Bootstrap starting.
I20260923 17:28:43.279944 13859 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:43.281121 13859 log.cc:824] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:43.284333 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.271466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52988 (local address 127.8.29.254:35781)
0923 17:28:43.272484 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.272496 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.272548 (+ 52us) server_negotiation.cc:407] Connection header received
0923 17:28:43.274945 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.275053 (+ 108us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.275604 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.275966 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:43.276631 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.277994 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.279757 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.280510 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.280657 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.281870 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:43.281880 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:43.281900 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:28:43.282608 (+ 708us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:43.282689 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:28:43.282713 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:43.282813 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:43.283036 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 17:28:43.283351 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":330,"threads_started":1}
I20260923 17:28:43.284515 13859 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e: No bootstrap required, opened a new log
I20260923 17:28:43.288601 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:28:43.293181 13859 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c594a576f428b974878a194c4a49e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35781 } }
I20260923 17:28:43.293588 13859 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:43.293682 13859 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2c594a576f428b974878a194c4a49e, State: Initialized, Role: FOLLOWER
I20260923 17:28:43.294261 13859 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c594a576f428b974878a194c4a49e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35781 } }
I20260923 17:28:43.294503 13859 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:43.294639 13859 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:43.294787 13859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:43.297154 13859 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c594a576f428b974878a194c4a49e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35781 } }
I20260923 17:28:43.297701 13859 leader_election.cc:302] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f2c594a576f428b974878a194c4a49e; no voters:
I20260923 17:28:43.298308 13859 leader_election.cc:288] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:43.298442 13864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:43.299187 13864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [term 1 LEADER]: Becoming Leader. State: Replica: 5f2c594a576f428b974878a194c4a49e, State: Running, Role: LEADER
I20260923 17:28:43.299787 13864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c594a576f428b974878a194c4a49e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35781 } }
I20260923 17:28:43.300522 13859 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:43.302671 13865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f2c594a576f428b974878a194c4a49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c594a576f428b974878a194c4a49e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35781 } } }
I20260923 17:28:43.303117 13865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:43.303663 13866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f2c594a576f428b974878a194c4a49e. Latest consensus state: current_term: 1 leader_uuid: "5f2c594a576f428b974878a194c4a49e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f2c594a576f428b974878a194c4a49e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35781 } } }
I20260923 17:28:43.303964 13873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:43.303996 13866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:43.309442 13873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:43.319070 13873 catalog_manager.cc:1347] Generated new cluster ID: b4d16ea422404227a049715e7d34c4cf
I20260923 17:28:43.319199 13873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:43.335798 13873 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:43.337118 13873 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:43.343039 13873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f2c594a576f428b974878a194c4a49e: Generated new TSK 0
I20260923 17:28:43.343878 13873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:43.849627 13862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:43.364521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52998 (local address 127.8.29.254:35781)
0923 17:28:43.364738 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:43.364745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:43.364932 (+ 187us) server_negotiation.cc:407] Connection header received
0923 17:28:43.365176 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:43.365183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:43.365286 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:43.365476 (+ 190us) server_negotiation.cc:226] Negotiated authn=JWT
0923 17:28:43.365910 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.367083 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.368486 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.368956 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.369070 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.369556 (+ 486us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 17:28:43.848842 (+479286us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:28:43.849291 (+ 449us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52998: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 17:28:43.849769 13862 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52998: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 17:28:43.849928 13888 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:43.364188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35781 (local address 127.0.0.1:52998)
0923 17:28:43.364842 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:43.364858 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:43.364983 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:43.365476 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:43.365479 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:43.365488 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
0923 17:28:43.365767 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.365775 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.367234 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.367238 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.368323 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:43.368334 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:43.369133 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:43.369139 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:43.369275 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:43.369311 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 17:28:43.849180 (+479869us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 17:28:43.849493 (+ 313us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.29.254:35781: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":426,"spinlock_wait_cycles":29952,"thread_start_us":400,"threads_started":1}
I20260923 17:28:43.851302 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 13793
I20260923 17:28:43.852495 13854 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:43.953650 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 13793
2026-09-23T17:28:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (1192 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (405 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (409 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:44 dist-test-slave-p7jt krb5kdc[13903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:44 dist-test-slave-p7jt krb5kdc[13903](info): set up 2 sockets
Sep 23 17:28:44 dist-test-slave-p7jt krb5kdc[13903](info): commencing operation
krb5kdc: starting...
W20260923 17:28:45.807010 8311 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:45 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:28:46.010836 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:34341
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:34341
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:46.332264 13916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:46.332589 13916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:46.332738 13916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:46.340097 13916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:46.340227 13916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:46.340312 13916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:46.340402 13916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:46.340463 13916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:46.350505 13916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:34341
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:34341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:46.351869 13916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:46.353207 13916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:46.364945 13924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.364982 13922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.365011 13921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:46.365712 13916 server_base.cc:1034] running on GCE node
I20260923 17:28:46.366539 13916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:46.367978 13916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:46.369182 13916 hybrid_clock.cc:648] HybridClock initialized: now 1790184526369123 us; error 42 us; skew 500 ppm
I20260923 17:28:46.369553 13916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:46 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:46.372829 13916 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:28:46.374377 13916 webserver.cc:460] Webserver started at http://127.8.29.254:41261/ using document root <none> and password file <none>
I20260923 17:28:46.375116 13916 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:46.375238 13916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:46.375581 13916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:46.378859 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "c5213915ecc944fca592903777825077"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-p7jt"
server_key: "f2ea4eeca03b714dc6250d5c42c55059"
server_key_iv: "d3ee46840b69523f0980730066fd67df"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.379720 13916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "c5213915ecc944fca592903777825077"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-p7jt"
server_key: "f2ea4eeca03b714dc6250d5c42c55059"
server_key_iv: "d3ee46840b69523f0980730066fd67df"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.385248 13916 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:46.388998 13931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.390525 13916 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:28:46.390709 13916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "c5213915ecc944fca592903777825077"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-p7jt"
server_key: "f2ea4eeca03b714dc6250d5c42c55059"
server_key_iv: "d3ee46840b69523f0980730066fd67df"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.390908 13916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:46.416123 13916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:46.420634 13916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:46.420907 13916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:46.438444 13916 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:34341
I20260923 17:28:46.438503 13982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:34341 every 8 connection(s)
I20260923 17:28:46.439857 13916 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:46.440497 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 13916
I20260923 17:28:46.440764 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:46.441648 8311 external_mini_cluster.cc:1428] Setting key d8c064c68a115b67ec0f277668ef7a73
I20260923 17:28:46.444814 13983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:28:46 dist-test-slave-p7jt krb5kdc[13903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:46.457314 13983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077: Bootstrap starting.
I20260923 17:28:46.461520 13983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:46.462728 13983 log.cc:824] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:46.465984 13983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077: No bootstrap required, opened a new log
I20260923 17:28:46.466279 13986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:46.443790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.8.29.254:34341)
0923 17:28:46.444690 (+ 900us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:46.444707 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:46.444760 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:28:46.447063 (+ 2303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:46.447091 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:46.447587 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:46.448072 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:46.449377 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.451429 (+ 2052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.452785 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:46.453770 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:46.453946 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:46.457707 (+ 3761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:46.457722 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:46.457745 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:46.457798 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:46.461731 (+ 3933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.462526 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.462535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.462544 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.462771 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:46.463193 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:46.463198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:46.463202 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:46.463849 (+ 647us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:46.464112 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:46.464535 (+ 423us) server_negotiation.cc:299] Negotiation successful
0923 17:28:46.464961 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":650,"thread_start_us":417,"threads_started":1}
I20260923 17:28:46.473465 13983 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5213915ecc944fca592903777825077" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34341 } }
I20260923 17:28:46.473830 13983 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:46.473902 13983 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5213915ecc944fca592903777825077, State: Initialized, Role: FOLLOWER
I20260923 17:28:46.474532 13983 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5213915ecc944fca592903777825077" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34341 } }
I20260923 17:28:46.474740 13983 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:46.474838 13983 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:46.474967 13983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:46.477065 13983 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5213915ecc944fca592903777825077" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34341 } }
I20260923 17:28:46.477541 13983 leader_election.cc:302] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5213915ecc944fca592903777825077; no voters:
I20260923 17:28:46.478435 13983 leader_election.cc:288] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:46.478524 13988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:46.479255 13988 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [term 1 LEADER]: Becoming Leader. State: Replica: c5213915ecc944fca592903777825077, State: Running, Role: LEADER
I20260923 17:28:46.479858 13988 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5213915ecc944fca592903777825077" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34341 } }
I20260923 17:28:46.480757 13983 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:46.482626 13990 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5213915ecc944fca592903777825077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5213915ecc944fca592903777825077" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34341 } } }
I20260923 17:28:46.483059 13990 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:46.485426 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5213915ecc944fca592903777825077. Latest consensus state: current_term: 1 leader_uuid: "c5213915ecc944fca592903777825077" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5213915ecc944fca592903777825077" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34341 } } }
I20260923 17:28:46.485770 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:46.487638 13996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:46.491797 13996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:46.501000 13996 catalog_manager.cc:1347] Generated new cluster ID: 6294307166dc4462b84c0ceb53694efe
I20260923 17:28:46.501147 13996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:46.508872 13996 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:46.510854 13996 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:46.528891 13996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5213915ecc944fca592903777825077: Generated new TSK 0
I20260923 17:28:46.530068 13996 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:28:46.586638 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34341
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:46.907483 14009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:46.907773 14009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:46.907882 14009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:46.915252 14009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:46.915369 14009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:46.915522 14009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:46.925486 14009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:46.926792 14009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:46.928087 14009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:46.941527 14014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.941461 14015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:46.941527 14017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:46.942266 14009 server_base.cc:1034] running on GCE node
I20260923 17:28:46.943008 14009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:46.944054 14009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:46.945277 14009 hybrid_clock.cc:648] HybridClock initialized: now 1790184526945218 us; error 53 us; skew 500 ppm
I20260923 17:28:46.945597 14009 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:46 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:46.948813 14009 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:28:46.950327 14009 webserver.cc:460] Webserver started at http://127.8.29.193:38867/ using document root <none> and password file <none>
I20260923 17:28:46.951068 14009 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:46.951189 14009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:46.951494 14009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:46.954610 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "6790aead0f8a400fa34ba969b2147931"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-p7jt"
server_key: "a9d2345e98db80c728a311efb41822e3"
server_key_iv: "a6289f8e18a5270703dbe60672b45903"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.955415 14009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "6790aead0f8a400fa34ba969b2147931"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-p7jt"
server_key: "a9d2345e98db80c728a311efb41822e3"
server_key_iv: "a6289f8e18a5270703dbe60672b45903"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.961086 14009 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:46.965142 14024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.966950 14009 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:28:46.967144 14009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "6790aead0f8a400fa34ba969b2147931"
format_stamp: "Formatted at 2026-09-23 17:28:46 on dist-test-slave-p7jt"
server_key: "a9d2345e98db80c728a311efb41822e3"
server_key_iv: "a6289f8e18a5270703dbe60672b45903"
server_key_version: "encryptionkey@0"
I20260923 17:28:46.967353 14009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:46.989418 14009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:46.994102 14009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:46.994388 14009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:46.995487 14009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:46.996958 14009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:46.997044 14009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:46.997164 14009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:46.997216 14009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.028471 14009 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:46801
I20260923 17:28:47.028522 14136 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:46801 every 8 connection(s)
I20260923 17:28:47.029798 14009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:47.036067 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 14009
I20260923 17:28:47.036278 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:47.036877 8311 external_mini_cluster.cc:1428] Setting key 83f81e74b2f1aaed02893bc59e3208c9
Sep 23 17:28:47 dist-test-slave-p7jt krb5kdc[13903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184526, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:47.050109 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.032359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:58079 (local address 127.8.29.254:34341)
0923 17:28:47.032844 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:47.032849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:47.033857 (+ 1008us) server_negotiation.cc:407] Connection header received
0923 17:28:47.035716 (+ 1859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:47.035722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:47.035801 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:47.035893 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:47.037510 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.038344 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.040076 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.040667 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.040798 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.044174 (+ 3376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:47.044179 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:47.044183 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:47.044216 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:47.046747 (+ 2531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:47.047671 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:47.047675 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:47.047678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:47.047815 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:47.048313 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:47.048318 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:47.048320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:47.048608 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:47.048727 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:47.049602 (+ 875us) server_negotiation.cc:299] Negotiation successful
0923 17:28:47.049802 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":206,"threads_started":1}
I20260923 17:28:47.051471 14140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.032931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34341 (local address 127.8.29.193:58079)
0923 17:28:47.033683 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:47.033728 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:47.035425 (+ 1697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:47.036148 (+ 723us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:47.036158 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:47.036725 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:47.037326 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.037344 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.038573 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.038579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.039920 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.039934 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.040862 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.040867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.041071 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.041698 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:47.041722 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:47.043928 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:47.046904 (+ 2976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:47.046914 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:47.046933 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:47.047526 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:47.047915 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:47.047920 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:47.047923 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:47.048168 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:47.048732 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:47.048740 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:47.049181 (+ 441us) client_negotiation.cc:769] Sending connection context
0923 17:28:47.049467 (+ 286us) client_negotiation.cc:240] Negotiation successful
0923 17:28:47.050026 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":295,"threads_started":1}
I20260923 17:28:47.053292 14137 heartbeater.cc:347] Connected to a master server at 127.8.29.254:34341
I20260923 17:28:47.053617 14137 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:47.054435 14137 heartbeater.cc:511] Master 127.8.29.254:34341 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:28:47.056664 13948 ts_manager.cc:194] Registered new tserver with Master: 6790aead0f8a400fa34ba969b2147931 (127.8.29.193:46801)
I20260923 17:28:47.058180 13948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:58079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:28:47.082762 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34341
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:47.402601 14143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:47.402889 14143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:47.403043 14143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:47.410362 14143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:47.410518 14143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:47.410676 14143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:47.420614 14143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:47.421933 14143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:47.423318 14143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:47.436952 14148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:47.437076 14151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:47.437107 14149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:47.437743 14143 server_base.cc:1034] running on GCE node
I20260923 17:28:47.438395 14143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:47.439383 14143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:47.440591 14143 hybrid_clock.cc:648] HybridClock initialized: now 1790184527440521 us; error 66 us; skew 500 ppm
I20260923 17:28:47.440896 14143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:47 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:47.444139 14143 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:28:47.445674 14143 webserver.cc:460] Webserver started at http://127.8.29.194:42529/ using document root <none> and password file <none>
I20260923 17:28:47.446353 14143 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:47.446499 14143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:47.446838 14143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:47.449708 14143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "0d711e12b23749eaa911fc2268797b69"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-p7jt"
server_key: "754d9745f9a9eb9ad5bad9ca3a489fe9"
server_key_iv: "9aa8cb7e3c0a6df813ca1d4f98989417"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.450486 14143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "0d711e12b23749eaa911fc2268797b69"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-p7jt"
server_key: "754d9745f9a9eb9ad5bad9ca3a489fe9"
server_key_iv: "9aa8cb7e3c0a6df813ca1d4f98989417"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.455621 14143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:28:47.459064 14158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.460438 14143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:47.460613 14143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "0d711e12b23749eaa911fc2268797b69"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-p7jt"
server_key: "754d9745f9a9eb9ad5bad9ca3a489fe9"
server_key_iv: "9aa8cb7e3c0a6df813ca1d4f98989417"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.460821 14143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:47.496248 14143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:47.500530 14143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:47.500811 14143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:47.501822 14143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:47.503360 14143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:47.503438 14143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.503582 14143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:47.503645 14143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.533931 14143 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:42309
I20260923 17:28:47.534034 14270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:42309 every 8 connection(s)
I20260923 17:28:47.535351 14143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:47.542726 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 14143
I20260923 17:28:47.542936 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:47.543469 8311 external_mini_cluster.cc:1428] Setting key 5f67bd6fd383c1b0ff90f3e01062b5c3
Sep 23 17:28:47 dist-test-slave-p7jt krb5kdc[13903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:47.557793 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.538068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:46063 (local address 127.8.29.254:34341)
0923 17:28:47.538298 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:47.538302 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:47.539775 (+ 1473us) server_negotiation.cc:407] Connection header received
0923 17:28:47.541642 (+ 1867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:47.541648 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:47.541739 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:47.541828 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:47.543951 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.544870 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.546933 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.547411 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.547528 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.551558 (+ 4030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:47.551563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:47.551567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:47.551602 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:47.554352 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:47.555231 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:47.555236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:47.555239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:47.555398 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:47.555933 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:47.555937 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:47.555940 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:47.556249 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:47.556401 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:47.557193 (+ 792us) server_negotiation.cc:299] Negotiation successful
0923 17:28:47.557460 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 17:28:47.558887 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:47.538656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34341 (local address 127.8.29.194:46063)
0923 17:28:47.539585 (+ 929us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:47.539647 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:47.541334 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:47.542093 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:47.542102 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:47.542852 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:47.543728 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.543757 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.545028 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.545033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.546768 (+ 1735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:47.546783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:47.547644 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:47.547651 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:47.547941 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:47.548639 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:47.548667 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:47.551286 (+ 2619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:47.554537 (+ 3251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:47.554549 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:47.554568 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:47.555032 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:47.555552 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:47.555558 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:47.555561 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:47.555797 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:47.556384 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:47.556394 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:47.556795 (+ 401us) client_negotiation.cc:769] Sending connection context
0923 17:28:47.557048 (+ 253us) client_negotiation.cc:240] Negotiation successful
0923 17:28:47.557622 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":582,"thread_start_us":370,"threads_started":1}
I20260923 17:28:47.560361 14271 heartbeater.cc:347] Connected to a master server at 127.8.29.254:34341
I20260923 17:28:47.560653 14271 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:47.561411 14271 heartbeater.cc:511] Master 127.8.29.254:34341 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
I20260923 17:28:47.562888 13948 ts_manager.cc:194] Registered new tserver with Master: 0d711e12b23749eaa911fc2268797b69 (127.8.29.194:42309)
I20260923 17:28:47.563745 13948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:46063
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:28:47.590694 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34341
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:47.912086 14276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:47.912375 14276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:47.912503 14276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:47.919843 14276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:47.919961 14276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:47.920109 14276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:47.930038 14276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34341
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:47.931435 14276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:47.932788 14276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:47.946802 14281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:47.946892 14282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:47.947856 14276 server_base.cc:1034] running on GCE node
W20260923 17:28:47.947927 14284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:47.948563 14276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:47.949528 14276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:47.950775 14276 hybrid_clock.cc:648] HybridClock initialized: now 1790184527950708 us; error 56 us; skew 500 ppm
I20260923 17:28:47.951109 14276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:47 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:47.954236 14276 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:28:47.955708 14276 webserver.cc:460] Webserver started at http://127.8.29.195:40277/ using document root <none> and password file <none>
I20260923 17:28:47.956430 14276 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:47.956559 14276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:47.956846 14276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:47.959877 14276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "19e5adf307aa4029b816cf24630225fb"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-p7jt"
server_key: "aca0fcdc3fb17599cd8d8ebd866ee578"
server_key_iv: "df63dd90acd571726580b99bd0018a6e"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.960616 14276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "19e5adf307aa4029b816cf24630225fb"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-p7jt"
server_key: "aca0fcdc3fb17599cd8d8ebd866ee578"
server_key_iv: "df63dd90acd571726580b99bd0018a6e"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.965677 14276 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 17:28:47.969097 14291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:47.970531 14276 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:47.970678 14276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "19e5adf307aa4029b816cf24630225fb"
format_stamp: "Formatted at 2026-09-23 17:28:47 on dist-test-slave-p7jt"
server_key: "aca0fcdc3fb17599cd8d8ebd866ee578"
server_key_iv: "df63dd90acd571726580b99bd0018a6e"
server_key_version: "encryptionkey@0"
I20260923 17:28:47.970868 14276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:47.993989 14276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:47.998529 14276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:47.998833 14276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:47.999873 14276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:48.001262 14276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:48.001336 14276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:48.001417 14276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:48.001459 14276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:48.032419 14276 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:36863
I20260923 17:28:48.032457 14403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:36863 every 8 connection(s)
I20260923 17:28:48.033711 14276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:48.041023 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 14276
I20260923 17:28:48.041211 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:48.041724 8311 external_mini_cluster.cc:1428] Setting key 868ad6f6159b5fb3e7a7a497ac44cf52
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184527, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:28:48.053901 14139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.036226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:52309 (local address 127.8.29.254:34341)
0923 17:28:48.036402 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.036407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.037629 (+ 1222us) server_negotiation.cc:407] Connection header received
0923 17:28:48.039508 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.039513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.039591 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.039685 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.041448 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.042488 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.044275 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.044886 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.045016 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.048650 (+ 3634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.048655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.048659 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:48.048696 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.051121 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.051864 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.051867 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.051869 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.051996 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.052383 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.052386 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.052388 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.052666 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.052773 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.053477 (+ 704us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.053648 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:28:48.054996 14406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.036684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34341 (local address 127.8.29.195:52309)
0923 17:28:48.037460 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:48.037505 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:48.039213 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:48.039998 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:48.040008 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:48.040546 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:48.041240 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.041266 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.042666 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.042670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.044134 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:48.044148 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.045101 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.045107 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:48.045328 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.045996 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:48.046021 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:48.048413 (+ 2392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:48.051291 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:48.051299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:48.051315 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:48.051769 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:48.052107 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:48.052110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:48.052113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:48.052299 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:48.052783 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:48.052789 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:48.053162 (+ 373us) client_negotiation.cc:769] Sending connection context
0923 17:28:48.053404 (+ 242us) client_negotiation.cc:240] Negotiation successful
0923 17:28:48.053819 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":327,"threads_started":1}
I20260923 17:28:48.056659 14404 heartbeater.cc:347] Connected to a master server at 127.8.29.254:34341
I20260923 17:28:48.056989 14404 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:48.057814 14404 heartbeater.cc:511] Master 127.8.29.254:34341 requested a full tablet report, sending...
I20260923 17:28:48.059247 13947 ts_manager.cc:194] Registered new tserver with Master: 19e5adf307aa4029b816cf24630225fb (127.8.29.195:36863)
I20260923 17:28:48.060106 13947 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:52309
I20260923 17:28:48.061514 14137 heartbeater.cc:503] Master 127.8.29.254:34341 was elected leader, sending a full tablet report...
I20260923 17:28:48.070075 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/test-admin.keytab.
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:48.096185 8311 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 17:28:48.101305 14411 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:48.099001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51528 (local address 127.8.29.193:46801)
0923 17:28:48.099584 (+ 583us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.099591 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.099618 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:28:48.099684 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.099690 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.099897 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.100047 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.100872 (+ 825us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51528: BlockingRecv error: recv got EOF from 127.0.0.1:51528 (error 108)
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":287,"threads_started":1}
W20260923 17:28:48.101449 8311 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:48.103056 8311 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:48.104712 8311 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:48.120035 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.106527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51540 (local address 127.8.29.193:46801)
0923 17:28:48.106739 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.106745 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.106869 (+ 124us) server_negotiation.cc:407] Connection header received
0923 17:28:48.107061 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.107065 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.107132 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.107212 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.108190 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.109147 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.110334 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.110971 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.111082 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.114639 (+ 3557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.114651 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.114666 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:48.114705 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.117287 (+ 2582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.118079 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.118087 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.118089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.118233 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.118673 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.118677 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.118679 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.119066 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.119303 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.119558 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.119760 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 17:28:48.125842 14411 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:48.124158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51552 (local address 127.8.29.193:46801)
0923 17:28:48.124356 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.124362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.124574 (+ 212us) server_negotiation.cc:407] Connection header received
0923 17:28:48.124762 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.124767 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.124826 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.124907 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.125633 (+ 726us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51552: BlockingRecv error: recv got EOF from 127.0.0.1:51552 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 17:28:48.126276 8311 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:48.127719 8311 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:48.129034 8311 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:48.142267 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.130328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.8.29.193:46801)
0923 17:28:48.130539 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.130545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.130806 (+ 261us) server_negotiation.cc:407] Connection header received
0923 17:28:48.130965 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.130969 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.131068 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.131150 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.132068 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.132772 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.133972 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.134404 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.134535 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.137753 (+ 3218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.137758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.137761 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:48.137805 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.140155 (+ 2350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.140826 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.140829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.140832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.140947 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.141251 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.141254 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.141256 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.141515 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.141707 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.141900 (+ 193us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.142054 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260923 17:28:48.146543 14411 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:28:48.144870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.8.29.193:46801)
0923 17:28:48.145057 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.145062 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.145293 (+ 231us) server_negotiation.cc:407] Connection header received
0923 17:28:48.145435 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.145439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.145506 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.145584 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.146305 (+ 721us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51570: BlockingRecv error: recv got EOF from 127.0.0.1:51570 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260923 17:28:48.146718 8311 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790184478467572-8311-0/krb5kdc/krb5cc' of type FILE
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:48.148072 8311 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:48.149210 8311 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:28:48.162739 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.150491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51580 (local address 127.8.29.193:46801)
0923 17:28:48.150668 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.150673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.150801 (+ 128us) server_negotiation.cc:407] Connection header received
0923 17:28:48.150912 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.150916 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.150972 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.151087 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.151909 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.152646 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.153870 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.154292 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.154386 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.157848 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.157854 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.157857 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:48.157892 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.160410 (+ 2518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.161027 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.161054 (+ 27us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.161056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.161205 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.161497 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.161500 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.161502 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.161771 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.161992 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.162211 (+ 219us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.162385 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:28:48.164251 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 14009
I20260923 17:28:48.176209 14411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.164955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51592 (local address 127.8.29.193:46801)
0923 17:28:48.165131 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.165137 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.165450 (+ 313us) server_negotiation.cc:407] Connection header received
0923 17:28:48.165601 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.165606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.165671 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.165747 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.166675 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.167383 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.168548 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.168957 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.169069 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.171228 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.171233 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.171237 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:48.171268 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.173658 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.174278 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.174284 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.174287 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.174417 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.174810 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.174815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.174819 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.175109 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.175350 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.175687 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.175904 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:28:48.176761 14132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:48.259047 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 14009
I20260923 17:28:48.272507 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 14143
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:28:48.288813 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.273619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.8.29.194:42309)
0923 17:28:48.274141 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.274148 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.274173 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:28:48.274232 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.274238 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.274477 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.274627 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.275653 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.276709 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.278047 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.278716 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.278823 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.282775 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.282788 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.282804 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:48.282842 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.285839 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.286705 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.286713 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.286715 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.286845 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.287271 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.287276 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.287280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.287635 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.287911 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.288241 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.288492 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":242,"threads_started":1}
I20260923 17:28:48.290228 14266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:48.374627 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 14143
I20260923 17:28:48.387318 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 14276
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[13903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:28:48.402879 14426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:48.388292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.8.29.195:36863)
0923 17:28:48.388955 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:48.388963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:48.388993 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:28:48.389051 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:48.389058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:48.389267 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:48.389393 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:48.390374 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.391463 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.392650 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:48.393216 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:48.393311 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:48.397315 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:48.397328 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:48.397345 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:48.397386 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:48.400178 (+ 2792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.400886 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.400894 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.400896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.401063 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:48.401454 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:48.401457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:48.401460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:48.401817 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:48.402075 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:48.402323 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 17:28:48.402598 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":366,"threads_started":1}
I20260923 17:28:48.404161 14399 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:48.486759 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 14276
I20260923 17:28:48.499168 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 13916
I20260923 17:28:48.500401 13978 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:48.603329 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 13916
2026-09-23T17:28:48Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3838 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[14431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[14431](info): set up 2 sockets
Sep 23 17:28:48 dist-test-slave-p7jt krb5kdc[14431](info): commencing operation
krb5kdc: starting...
W20260923 17:28:49.607240 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.966s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:49 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:49Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.254.keytab.
Entry for principal oryx/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.254.keytab.
I20260923 17:28:49.815424 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:34369
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.254.keytab
--principal=oryx/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:34369
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:50.138863 14444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:50.139195 14444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:50.139333 14444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:50.146751 14444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:50.146890 14444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:50.146973 14444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:50.147056 14444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:50.147118 14444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:50.157245 14444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:34369
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.254.keytab
--principal=oryx/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:34369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:50.158674 14444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:50.160006 14444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:50.171913 14450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:50.171939 14452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:50.171977 14449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:50.173115 14444 server_base.cc:1034] running on GCE node
I20260923 17:28:50.173966 14444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:50.175513 14444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:50.176736 14444 hybrid_clock.cc:648] HybridClock initialized: now 1790184530176688 us; error 36 us; skew 500 ppm
I20260923 17:28:50.177134 14444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:50 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184530, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:50.180380 14444 init.cc:377] Logged in from keytab as oryx/127.8.29.254@KRBTEST.COM (short username oryx)
I20260923 17:28:50.181979 14444 webserver.cc:460] Webserver started at http://127.8.29.254:40945/ using document root <none> and password file <none>
I20260923 17:28:50.182740 14444 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:50.182860 14444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:50.183157 14444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:50.186255 14444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "6893904a76694629b825a55109145c87"
format_stamp: "Formatted at 2026-09-23 17:28:50 on dist-test-slave-p7jt"
server_key: "d66f614f8919ebe6d70711fcc34a1777"
server_key_iv: "7d490c16e2fc3c59885d65937a522c6d"
server_key_version: "encryptionkey@0"
I20260923 17:28:50.187101 14444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "6893904a76694629b825a55109145c87"
format_stamp: "Formatted at 2026-09-23 17:28:50 on dist-test-slave-p7jt"
server_key: "d66f614f8919ebe6d70711fcc34a1777"
server_key_iv: "7d490c16e2fc3c59885d65937a522c6d"
server_key_version: "encryptionkey@0"
I20260923 17:28:50.192628 14444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:50.196281 14459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:50.197687 14444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:28:50.197855 14444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "6893904a76694629b825a55109145c87"
format_stamp: "Formatted at 2026-09-23 17:28:50 on dist-test-slave-p7jt"
server_key: "d66f614f8919ebe6d70711fcc34a1777"
server_key_iv: "7d490c16e2fc3c59885d65937a522c6d"
server_key_version: "encryptionkey@0"
I20260923 17:28:50.198081 14444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:50.217799 14444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:50.222505 14444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:50.222788 14444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:50.240746 14444 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:34369
I20260923 17:28:50.240779 14510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:34369 every 8 connection(s)
I20260923 17:28:50.242110 14444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:50.245795 14511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:50.245815 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 14444
I20260923 17:28:50.246022 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:50.246673 8311 external_mini_cluster.cc:1428] Setting key fc454b65a333c1ccfd2d3bd6e9603d5d
I20260923 17:28:50.259703 14511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: Bootstrap starting.
Sep 23 17:28:50 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:28:50.262989 14511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:50.264331 14511 log.cc:824] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:50.267916 14511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: No bootstrap required, opened a new log
I20260923 17:28:50.271651 14514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:50.248690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49176 (local address 127.8.29.254:34369)
0923 17:28:50.249514 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:50.249530 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:50.249579 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:28:50.251956 (+ 2377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:50.251987 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:50.252542 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:50.253030 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:50.254374 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:50.256364 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:50.257692 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:50.258770 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:50.258944 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:50.262792 (+ 3848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:50.262803 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:50.262822 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:50.262876 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:50.266528 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:50.267354 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:50.267362 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:50.267371 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:50.267598 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:50.268073 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:50.268077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:50.268080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:50.268840 (+ 760us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:50.269154 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:50.269643 (+ 489us) server_negotiation.cc:299] Negotiation successful
0923 17:28:50.270157 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":303,"threads_started":1}
I20260923 17:28:50.277422 14511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } }
I20260923 17:28:50.277830 14511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:50.277922 14511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6893904a76694629b825a55109145c87, State: Initialized, Role: FOLLOWER
I20260923 17:28:50.278548 14511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } }
I20260923 17:28:50.278769 14511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:50.278898 14511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:50.279037 14511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:50.281315 14511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } }
I20260923 17:28:50.281790 14511 leader_election.cc:302] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6893904a76694629b825a55109145c87; no voters:
I20260923 17:28:50.282341 14511 leader_election.cc:288] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:50.282523 14516 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:50.283711 14516 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 1 LEADER]: Becoming Leader. State: Replica: 6893904a76694629b825a55109145c87, State: Running, Role: LEADER
I20260923 17:28:50.284268 14516 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } }
I20260923 17:28:50.284691 14511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:50.286836 14518 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6893904a76694629b825a55109145c87. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } }
I20260923 17:28:50.287171 14518 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:50.287602 14517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } }
I20260923 17:28:50.287900 14517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:50.288743 14523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:50.294407 14523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:50.304867 14523 catalog_manager.cc:1347] Generated new cluster ID: d7687137e2964f55b9c555ae17474c0a
I20260923 17:28:50.304983 14523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:50.322439 14523 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:50.324422 14523 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:50.341054 14523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: Generated new TSK 0
I20260923 17:28:50.342181 14523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab.
Entry for principal oryx/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab.
I20260923 17:28:50.398811 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34369
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab
--principal=oryx/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:50.728327 14537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:50.728636 14537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:50.728768 14537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:50.736007 14537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:50.736150 14537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:50.736294 14537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:50.746137 14537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab
--principal=oryx/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:50.747491 14537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:50.748888 14537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:50.761570 14543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:50.761540 14545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:50.761509 14542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:50.762279 14537 server_base.cc:1034] running on GCE node
I20260923 17:28:50.763007 14537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:50.764173 14537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:50.765378 14537 hybrid_clock.cc:648] HybridClock initialized: now 1790184530765326 us; error 45 us; skew 500 ppm
I20260923 17:28:50.765734 14537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:50 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184530, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:50.768883 14537 init.cc:377] Logged in from keytab as oryx/127.8.29.193@KRBTEST.COM (short username oryx)
I20260923 17:28:50.770305 14537 webserver.cc:460] Webserver started at http://127.8.29.193:34339/ using document root <none> and password file <none>
I20260923 17:28:50.771030 14537 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:50.771150 14537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:50.771466 14537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:50.774389 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "455cb603b43346f0b1d56e6dc06f589b"
format_stamp: "Formatted at 2026-09-23 17:28:50 on dist-test-slave-p7jt"
server_key: "63ca067b67201608d61476c61bda2adb"
server_key_iv: "ebbda28ad054523e059b4eb1b788ca60"
server_key_version: "encryptionkey@0"
I20260923 17:28:50.775168 14537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "455cb603b43346f0b1d56e6dc06f589b"
format_stamp: "Formatted at 2026-09-23 17:28:50 on dist-test-slave-p7jt"
server_key: "63ca067b67201608d61476c61bda2adb"
server_key_iv: "ebbda28ad054523e059b4eb1b788ca60"
server_key_version: "encryptionkey@0"
I20260923 17:28:50.780360 14537 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:50.783854 14552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:50.785347 14537 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:50.785497 14537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "455cb603b43346f0b1d56e6dc06f589b"
format_stamp: "Formatted at 2026-09-23 17:28:50 on dist-test-slave-p7jt"
server_key: "63ca067b67201608d61476c61bda2adb"
server_key_iv: "ebbda28ad054523e059b4eb1b788ca60"
server_key_version: "encryptionkey@0"
I20260923 17:28:50.785683 14537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:50.804397 14537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:50.808269 14537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:50.808537 14537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:50.809473 14537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:50.810920 14537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:50.810995 14537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:50.811120 14537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:50.811170 14537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:50.840615 14537 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:35875
I20260923 17:28:50.840726 14664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:35875 every 8 connection(s)
I20260923 17:28:50.842201 14537 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:50.850814 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 14537
I20260923 17:28:50.851023 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:28:50.851619 8311 external_mini_cluster.cc:1428] Setting key 49e02c514d0a3c22fc3e5cec31f000f1
Sep 23 17:28:50 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184530, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:28:50.861110 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:50.844725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:50575 (local address 127.8.29.254:34369)
0923 17:28:50.845107 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:50.845112 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:50.846008 (+ 896us) server_negotiation.cc:407] Connection header received
0923 17:28:50.847828 (+ 1820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:50.847834 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:50.847920 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:50.848016 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:50.849512 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:50.850223 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:50.851843 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:50.852235 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:50.852344 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:50.855703 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:50.855709 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:50.855713 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:50.855747 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:50.858118 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:50.858876 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:50.858880 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:50.858882 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:50.859007 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:50.859462 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:50.859466 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:50.859468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:50.859744 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:50.859861 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:50.860679 (+ 818us) server_negotiation.cc:299] Negotiation successful
0923 17:28:50.860855 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":187,"threads_started":1}
I20260923 17:28:50.862505 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:50.845194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.193:50575)
0923 17:28:50.845856 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:50.845901 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:50.847549 (+ 1648us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:50.848255 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:50.848264 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:50.848762 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:50.849345 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:50.849362 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:50.850366 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:50.850372 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:50.851701 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:50.851716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:50.852448 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:50.852456 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:50.852691 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:50.853303 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:50.853330 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:50.855434 (+ 2104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:50.858255 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:50.858264 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:50.858284 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:50.858753 (+ 469us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:50.859106 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:50.859111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:50.859115 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:50.859340 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:50.859860 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:50.859868 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:50.860310 (+ 442us) client_negotiation.cc:769] Sending connection context
0923 17:28:50.860602 (+ 292us) client_negotiation.cc:240] Negotiation successful
0923 17:28:50.861159 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":262,"threads_started":1}
I20260923 17:28:50.864023 14665 heartbeater.cc:347] Connected to a master server at 127.8.29.254:34369
I20260923 17:28:50.864341 14665 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:50.865234 14665 heartbeater.cc:511] Master 127.8.29.254:34369 requested a full tablet report, sending...
I20260923 17:28:50.867489 14476 ts_manager.cc:194] Registered new tserver with Master: 455cb603b43346f0b1d56e6dc06f589b (127.8.29.193:35875)
I20260923 17:28:50.868971 14476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:50575
WARNING: no policy specified for oryx/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab.
Entry for principal oryx/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab.
I20260923 17:28:50.897750 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34369
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab
--principal=oryx/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:51.220551 14671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:51.220839 14671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:51.220938 14671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:51.228296 14671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:51.228421 14671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:51.228533 14671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:51.238291 14671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab
--principal=oryx/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:51.239598 14671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:51.240939 14671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:51.253813 14676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:51.254539 14677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:51.254694 14679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:51.257891 14671 server_base.cc:1034] running on GCE node
I20260923 17:28:51.258651 14671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:51.259753 14671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:51.260977 14671 hybrid_clock.cc:648] HybridClock initialized: now 1790184531260915 us; error 63 us; skew 500 ppm
I20260923 17:28:51.261323 14671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:51 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184531, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:51.264605 14671 init.cc:377] Logged in from keytab as oryx/127.8.29.194@KRBTEST.COM (short username oryx)
I20260923 17:28:51.266258 14671 webserver.cc:460] Webserver started at http://127.8.29.194:46555/ using document root <none> and password file <none>
I20260923 17:28:51.267091 14671 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:51.267212 14671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:51.267505 14671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:51.270586 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "d7c0e000eae64596a8d6eb329c4598b9"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-p7jt"
server_key: "015ca976908037e18eaeffefd22d5a19"
server_key_iv: "44465d0ccc60948360057cd1ab065da4"
server_key_version: "encryptionkey@0"
I20260923 17:28:51.271354 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "d7c0e000eae64596a8d6eb329c4598b9"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-p7jt"
server_key: "015ca976908037e18eaeffefd22d5a19"
server_key_iv: "44465d0ccc60948360057cd1ab065da4"
server_key_version: "encryptionkey@0"
I20260923 17:28:51.276752 14671 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:28:51.280575 14686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:51.281962 14671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:51.282173 14671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "d7c0e000eae64596a8d6eb329c4598b9"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-p7jt"
server_key: "015ca976908037e18eaeffefd22d5a19"
server_key_iv: "44465d0ccc60948360057cd1ab065da4"
server_key_version: "encryptionkey@0"
I20260923 17:28:51.282373 14671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:51.299144 14671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:51.303587 14671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:51.303877 14671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:51.304920 14671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:51.306327 14671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:51.306403 14671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:51.306548 14671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:51.306602 14671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:51.337514 14671 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:34613
I20260923 17:28:51.337548 14798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:34613 every 8 connection(s)
I20260923 17:28:51.338783 14671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:51.348255 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 14671
I20260923 17:28:51.348450 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:28:51.348855 8311 external_mini_cluster.cc:1428] Setting key 2b76835cbaaa1dcba484d5c5f8077033
Sep 23 17:28:51 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184531, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:28:51.358553 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:51.341261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:53261 (local address 127.8.29.254:34369)
0923 17:28:51.341484 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:51.341490 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:51.342523 (+ 1033us) server_negotiation.cc:407] Connection header received
0923 17:28:51.344309 (+ 1786us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:51.344313 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:51.344383 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:51.344468 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:51.345882 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.346640 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.348295 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.348769 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.348861 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:51.352782 (+ 3921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:51.352786 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:51.352790 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:51.352822 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:51.355418 (+ 2596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:51.356199 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:51.356203 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:51.356205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:51.356323 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:51.356732 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:51.356735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:51.356736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:51.357056 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:51.357193 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:51.358077 (+ 884us) server_negotiation.cc:299] Negotiation successful
0923 17:28:51.358258 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:28:51.359537 14801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:51.341681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.194:53261)
0923 17:28:51.342354 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:51.342392 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:51.344053 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:51.344686 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:51.344695 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:51.345184 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:51.345745 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:51.345762 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.346768 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.346772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:51.348159 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:51.348173 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.348994 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.349003 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:51.349328 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:51.350046 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:51.350074 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:51.352533 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:51.355547 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:51.355563 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:51.355581 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:51.356095 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:51.356407 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:51.356411 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:51.356414 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:51.356642 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:51.357189 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:51.357198 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:51.357662 (+ 464us) client_negotiation.cc:769] Sending connection context
0923 17:28:51.357985 (+ 323us) client_negotiation.cc:240] Negotiation successful
0923 17:28:51.358406 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":245,"threads_started":1}
I20260923 17:28:51.360958 14799 heartbeater.cc:347] Connected to a master server at 127.8.29.254:34369
I20260923 17:28:51.361241 14799 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:51.361948 14799 heartbeater.cc:511] Master 127.8.29.254:34369 requested a full tablet report, sending...
I20260923 17:28:51.363489 14476 ts_manager.cc:194] Registered new tserver with Master: d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613)
I20260923 17:28:51.364307 14476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:53261
WARNING: no policy specified for oryx/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab.
Entry for principal oryx/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab.
I20260923 17:28:51.393751 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34369
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab
--principal=oryx/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:51.710223 14804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:51.710561 14804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:51.710688 14804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:51.718053 14804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:51.718158 14804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:51.718269 14804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:51.728298 14804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab
--principal=oryx/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:34369
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:51.729574 14804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:51.730952 14804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:51.744140 14812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:51.744123 14809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:51.744180 14810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:51.744947 14804 server_base.cc:1034] running on GCE node
I20260923 17:28:51.745553 14804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:51.746546 14804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:51.747771 14804 hybrid_clock.cc:648] HybridClock initialized: now 1790184531747722 us; error 55 us; skew 500 ppm
I20260923 17:28:51.748093 14804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:51 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184531, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:51.755517 14804 init.cc:377] Logged in from keytab as oryx/127.8.29.195@KRBTEST.COM (short username oryx)
I20260923 17:28:51.757167 14804 webserver.cc:460] Webserver started at http://127.8.29.195:44637/ using document root <none> and password file <none>
I20260923 17:28:51.758047 14804 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:51.758292 14804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:51.758692 14804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:51.761672 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "b6997baeee22454d9624893c37bca7cf"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-p7jt"
server_key: "8e82b902c09c5d79219edb53559a6d3f"
server_key_iv: "4139fde87b2fba281f171fda2be93592"
server_key_version: "encryptionkey@0"
I20260923 17:28:51.762408 14804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "b6997baeee22454d9624893c37bca7cf"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-p7jt"
server_key: "8e82b902c09c5d79219edb53559a6d3f"
server_key_iv: "4139fde87b2fba281f171fda2be93592"
server_key_version: "encryptionkey@0"
I20260923 17:28:51.768476 14804 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260923 17:28:51.772379 14819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:51.774178 14804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:28:51.774422 14804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "b6997baeee22454d9624893c37bca7cf"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-p7jt"
server_key: "8e82b902c09c5d79219edb53559a6d3f"
server_key_iv: "4139fde87b2fba281f171fda2be93592"
server_key_version: "encryptionkey@0"
I20260923 17:28:51.774712 14804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:51.802675 14804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:51.807785 14804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:51.808100 14804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:51.809288 14804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:51.811230 14804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:51.811353 14804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:51.811484 14804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:51.811546 14804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:51.847922 14804 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:44201
I20260923 17:28:51.847970 14931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:44201 every 8 connection(s)
I20260923 17:28:51.849217 14804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:51.853488 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 14804
I20260923 17:28:51.853677 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:28:51.854084 8311 external_mini_cluster.cc:1428] Setting key a4a89328eab677530bb4f1797fb04715
Sep 23 17:28:51 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184531, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:28:51.870306 14667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:51.851760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:43637 (local address 127.8.29.254:34369)
0923 17:28:51.851930 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:51.851934 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:51.853157 (+ 1223us) server_negotiation.cc:407] Connection header received
0923 17:28:51.855143 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:51.855148 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:51.855232 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:51.855332 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:51.857150 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.858015 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.859621 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.860077 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.860188 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:51.864021 (+ 3833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:51.864027 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:51.864030 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:51.864073 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:51.866889 (+ 2816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:51.867824 (+ 935us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:51.867829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:51.867832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:51.868027 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:51.868506 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:51.868511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:51.868513 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:51.868811 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:51.868962 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:51.869724 (+ 762us) server_negotiation.cc:299] Negotiation successful
0923 17:28:51.869960 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:28:51.871393 14934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:51.852223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.195:43637)
0923 17:28:51.852978 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:51.853026 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:51.854872 (+ 1846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:51.855625 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:51.855637 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:51.856317 (+ 680us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:51.856970 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:51.856991 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.858180 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.858183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:51.859458 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:51.859470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:51.860283 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:51.860288 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:51.860492 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:51.861171 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:51.861193 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:51.863682 (+ 2489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:51.867089 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:51.867098 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:51.867113 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:51.867641 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:51.868155 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:51.868161 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:51.868165 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:51.868380 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:51.868958 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:51.868968 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:51.869330 (+ 362us) client_negotiation.cc:769] Sending connection context
0923 17:28:51.869591 (+ 261us) client_negotiation.cc:240] Negotiation successful
0923 17:28:51.870172 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":274,"threads_started":1}
I20260923 17:28:51.872375 14665 heartbeater.cc:503] Master 127.8.29.254:34369 was elected leader, sending a full tablet report...
I20260923 17:28:51.872895 14932 heartbeater.cc:347] Connected to a master server at 127.8.29.254:34369
I20260923 17:28:51.873212 14932 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:51.874073 14932 heartbeater.cc:511] Master 127.8.29.254:34369 requested a full tablet report, sending...
I20260923 17:28:51.875665 14476 ts_manager.cc:194] Registered new tserver with Master: b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201)
I20260923 17:28:51.876492 14476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:43637
I20260923 17:28:51.882074 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.29.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.253.keytab.
Entry for principal oryx/127.8.29.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.253.keytab.
I20260923 17:28:51.928308 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.253:41173
--webserver_interface=127.8.29.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.253.keytab
--principal=oryx/127.8.29.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:34369,127.8.29.253:41173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:52.254011 14937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:52.254318 14937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:52.254426 14937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:52.261838 14937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:52.261968 14937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:52.262018 14937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:52.262120 14937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:52.262183 14937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:52.272771 14937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:34369,127.8.29.253:41173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.253.keytab
--principal=oryx/127.8.29.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.253:41173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.29.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
W20260923 17:28:52.273154 14937 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.29.254:34369,127.8.29.253:41173'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 17:28:52.274202 14937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:52.275557 14937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:52.287817 14945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:52.287849 14942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:52.287849 14943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:52.288504 14937 server_base.cc:1034] running on GCE node
I20260923 17:28:52.289642 14937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:52.291257 14937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:52.292526 14937 hybrid_clock.cc:648] HybridClock initialized: now 1790184532292451 us; error 41 us; skew 500 ppm
I20260923 17:28:52.292912 14937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:52 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:52.296032 14937 init.cc:377] Logged in from keytab as oryx/127.8.29.253@KRBTEST.COM (short username oryx)
I20260923 17:28:52.297508 14937 webserver.cc:460] Webserver started at http://127.8.29.253:39767/ using document root <none> and password file <none>
I20260923 17:28:52.298194 14937 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:52.298280 14937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:52.298599 14937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:52.301733 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data/instance:
uuid: "b894fbfff63a4a1297abadee53713db3"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "42b00c6e55f25bbdc3c632d95aa46df0"
server_key_iv: "4fc9f317257316ba84630dd6efa0c968"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.302608 14937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal/instance:
uuid: "b894fbfff63a4a1297abadee53713db3"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "42b00c6e55f25bbdc3c632d95aa46df0"
server_key_iv: "4fc9f317257316ba84630dd6efa0c968"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.308075 14937 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:28:52.311580 14952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:52.312978 14937 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:28:52.313153 14937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal
uuid: "b894fbfff63a4a1297abadee53713db3"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "42b00c6e55f25bbdc3c632d95aa46df0"
server_key_iv: "4fc9f317257316ba84630dd6efa0c968"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.313333 14937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:52.337754 14937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:52.342002 14937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:52.342269 14937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:52.359277 14937 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.253:41173
I20260923 17:28:52.359331 15003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.253:41173 every 8 connection(s)
I20260923 17:28:52.360584 14937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
I20260923 17:28:52.364152 15004 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:52.367007 14799 heartbeater.cc:503] Master 127.8.29.254:34369 was elected leader, sending a full tablet report...
I20260923 17:28:52.368198 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 14937
I20260923 17:28:52.371711 15004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:28:52 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.253@KRBTEST.COM
Sep 23 17:28:52 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.253@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:28:52.393042 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.375785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.8.29.254:34369)
0923 17:28:52.376329 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.376335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.376815 (+ 480us) server_negotiation.cc:407] Connection header received
0923 17:28:52.377000 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.377006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.377091 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.377209 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:52.378765 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.379824 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.381813 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.382288 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.382395 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.386817 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:52.386826 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:52.386831 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:52.386876 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:52.389866 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.390670 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.390674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.390676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.390840 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.391254 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.391259 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.391262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.391685 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:52.391829 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.392429 (+ 600us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.392685 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":246,"threads_started":1}
I20260923 17:28:52.393935 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.375784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:49192)
0923 17:28:52.376677 (+ 893us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:52.376721 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:52.376868 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:52.377273 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:52.377278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:52.378158 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:52.378561 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.378583 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.379986 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.379990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.381628 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.381645 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.382651 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.382664 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.382902 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.383586 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:52.383627 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:52.386546 (+ 2919us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:52.390007 (+ 3461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.390018 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.390023 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.390541 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.390980 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.390984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.390986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.391125 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.391827 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:52.391845 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:52.392077 (+ 232us) client_negotiation.cc:769] Sending connection context
0923 17:28:52.392300 (+ 223us) client_negotiation.cc:240] Negotiation successful
0923 17:28:52.392734 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":41,"thread_start_us":483,"threads_started":1}
I20260923 17:28:52.393953 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.370574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43430 (local address 127.8.29.253:41173)
0923 17:28:52.371387 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.371402 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.371442 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:28:52.373091 (+ 1649us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.373115 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.373562 (+ 447us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.373949 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:52.375513 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.377229 (+ 1716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.379089 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.380160 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.380314 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.385348 (+ 5034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:52.385361 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:52.385381 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:52.385429 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:52.389116 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.389999 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.390009 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.390020 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.390275 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.390751 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.390756 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.390759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.391496 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:52.391819 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.392167 (+ 348us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.392861 (+ 694us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":245,"server-negotiator.queue_time_us":612,"spinlock_wait_cycles":22272,"thread_start_us":411,"threads_started":1}
I20260923 17:28:52.395826 15004 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:28:52 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.253@KRBTEST.COM for oryx/127.8.29.253@KRBTEST.COM
I20260923 17:28:52.407121 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.396388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.0.0.1:43436)
0923 17:28:52.396544 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:52.396560 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:52.396654 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:52.396943 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:52.396947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:52.397263 (+ 316us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:52.397396 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.397403 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.398319 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.398322 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.399226 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.399237 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.399909 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.399913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.400072 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.400542 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:52.400558 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:52.402388 (+ 1830us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:52.404983 (+ 2595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.404988 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.404993 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.405332 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.405637 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.405641 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.405645 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.405757 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.406241 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:52.406246 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:52.406365 (+ 119us) client_negotiation.cc:769] Sending connection context
0923 17:28:52.406503 (+ 138us) client_negotiation.cc:240] Negotiation successful
0923 17:28:52.406772 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 17:28:52.407274 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.396566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43436 (local address 127.8.29.253:41173)
0923 17:28:52.396700 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.396705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.396720 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:28:52.396758 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.396762 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.396823 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.396908 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:52.397504 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.398208 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.399337 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.399787 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.399875 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.402562 (+ 2687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:52.402567 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:52.402571 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:52.402600 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:52.404865 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.405428 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.405433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.405436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.405544 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.405846 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.405850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.405854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.406125 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:52.406247 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.406593 (+ 346us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.406914 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":126,"server-negotiator.queue_time_us":48}
I20260923 17:28:52.411820 15004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Bootstrap starting.
I20260923 17:28:52.414857 15004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:52.415829 15004 log.cc:824] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:52.418479 15004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: No bootstrap required, opened a new log
I20260923 17:28:52.425561 15004 raft_consensus.cc:348] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } }
I20260923 17:28:52.425936 15004 raft_consensus.cc:374] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:52.426007 15004 raft_consensus.cc:729] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b894fbfff63a4a1297abadee53713db3, State: Initialized, Role: FOLLOWER
I20260923 17:28:52.426644 15004 consensus_queue.cc:260] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } }
I20260923 17:28:52.427495 15011 sys_catalog.cc:455] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } } }
I20260923 17:28:52.427819 15011 sys_catalog.cc:458] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:28:52.428469 15004 sys_catalog.cc:564] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:28:52.436995 15022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:28:52.437218 15022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:28:52.439451 14937 master_runner.cc:428] Detected that this master b894fbfff63a4a1297abadee53713db3 is joining an existing cluster
I20260923 17:28:52.439530 14937 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.29.253:41173
I20260923 17:28:52.457883 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.443021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49200 (local address 127.8.29.254:34369)
0923 17:28:52.443176 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.443180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.443816 (+ 636us) server_negotiation.cc:407] Connection header received
0923 17:28:52.444021 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.444027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.444126 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.444198 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:52.445347 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.446035 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.448346 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.448951 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.449050 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.451532 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:52.451538 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:52.451543 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:52.451594 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:52.454768 (+ 3174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.455515 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.455521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.455524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.455691 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.456119 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.456124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.456128 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.456464 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:52.456593 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.457278 (+ 685us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.457500 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:28:52.457976 15028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.443105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:49200)
0923 17:28:52.443715 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:52.443730 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:52.443866 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:52.444407 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:52.444412 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:52.444918 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:52.445203 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.445212 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.446163 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.446169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.448178 (+ 2009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.448191 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.449342 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.449348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.449518 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.450200 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:52.450221 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:52.451331 (+ 1110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:52.454918 (+ 3587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.454922 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.454926 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.455381 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.455829 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.455833 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.455836 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.455992 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.456696 (+ 704us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:52.456701 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:52.456900 (+ 199us) client_negotiation.cc:769] Sending connection context
0923 17:28:52.457163 (+ 263us) client_negotiation.cc:240] Negotiation successful
0923 17:28:52.457506 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":322,"threads_started":1}
I20260923 17:28:52.462260 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.444236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43444 (local address 127.8.29.253:41173)
0923 17:28:52.444388 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.444393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.446737 (+ 2344us) server_negotiation.cc:407] Connection header received
0923 17:28:52.446882 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.446887 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.446972 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.447066 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:52.448403 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.449478 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.451346 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.452051 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.452147 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.454643 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:52.454650 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:52.454655 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:52.454693 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:52.459068 (+ 4375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.459892 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.459901 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.459905 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.460038 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.460457 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.460462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.460465 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.460830 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:52.460985 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.461648 (+ 663us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.461883 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:28:52.463511 15029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.443880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.0.0.1:43444)
0923 17:28:52.446631 (+ 2751us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:52.446649 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:52.446760 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:52.447213 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:52.447218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:52.447917 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:52.448261 (+ 344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.448273 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.449697 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.449702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.451180 (+ 1478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.451195 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.452433 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.452448 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.452627 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.453194 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:52.453217 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:52.454320 (+ 1103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:52.459231 (+ 4911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.459236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.459240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.459774 (+ 534us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.460182 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.460187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.460191 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.460345 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.461071 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:52.461075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:52.461347 (+ 272us) client_negotiation.cc:769] Sending connection context
0923 17:28:52.462723 (+ 1376us) client_negotiation.cc:240] Negotiation successful
0923 17:28:52.463049 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2617,"thread_start_us":2405,"threads_started":1}
WARNING: no policy specified for oryx/127.8.29.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.252@KRBTEST.COM" created.
Sep 23 17:28:52 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184530, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.254@KRBTEST.COM for oryx/127.8.29.253@KRBTEST.COM
I20260923 17:28:52.490509 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.464522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.0.0.1:43452)
0923 17:28:52.473410 (+ 8888us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:52.473468 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:52.473783 (+ 315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:52.474409 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:52.474418 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:52.475439 (+ 1021us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:52.475750 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.475765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.477247 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.477253 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.479090 (+ 1837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.479111 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.480023 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.480031 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.480209 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.480970 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:52.480998 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:52.484038 (+ 3040us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:52.487765 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.487779 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.487787 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.488212 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.488592 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.488598 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.488602 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.488787 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.489324 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:52.489333 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:52.489524 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 17:28:52.489677 (+ 153us) client_negotiation.cc:240] Negotiation successful
0923 17:28:52.490027 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8586,"thread_start_us":8429,"threads_started":1}
I20260923 17:28:52.490442 15007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.464888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43452 (local address 127.8.29.253:41173)
0923 17:28:52.465391 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.465397 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.473607 (+ 8210us) server_negotiation.cc:407] Connection header received
0923 17:28:52.474070 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.474076 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.474205 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.474326 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:52.475965 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.477103 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.479268 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.479802 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.479933 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.484349 (+ 4416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:52.484359 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:52.484363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:52.484404 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:52.487620 (+ 3216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.488342 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.488348 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.488350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.488483 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.488907 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.488910 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.488913 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.489189 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:52.489323 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.489860 (+ 537us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.490115 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387}
I20260923 17:28:52.491086 15011 raft_consensus.cc:482] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:52.491473 15011 raft_consensus.cc:504] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } }
I20260923 17:28:52.492954 15011 leader_election.cc:288] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6893904a76694629b825a55109145c87 (127.8.29.254:34369)
I20260923 17:28:52.493364 14476 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.29.253:41173: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b894fbfff63a4a1297abadee53713db3"
member_type: NON_VOTER
last_known_addr {
host: "127.8.29.253"
port: 41173
}
attrs {
promote: true
}
}
dest_uuid: "6893904a76694629b825a55109145c87"
cas_config_opid_index: -1
I20260923 17:28:52.493652 14486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b894fbfff63a4a1297abadee53713db3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6893904a76694629b825a55109145c87" is_pre_election: true
I20260923 17:28:52.494791 14953 leader_election.cc:302] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: b894fbfff63a4a1297abadee53713db3; no voters: 6893904a76694629b825a55109145c87
I20260923 17:28:52.494735 14476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } }
I20260923 17:28:52.495446 15011 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 17:28:52.505653 15034 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER b894fbfff63a4a1297abadee53713db3 (127.8.29.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } } }
I20260923 17:28:52.506677 14979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:52.510322 14979 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Refusing update from remote peer 6893904a76694629b825a55109145c87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 17:28:52.511205 15034 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.29.253:41173
I20260923 17:28:52.512051 14937 master.cc:537] Master@127.8.29.253:41173 shutting down...
I20260923 17:28:52.511844 15036 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:52.512810 15035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } } }
I20260923 17:28:52.513157 15035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.252.keytab.
Entry for principal oryx/127.8.29.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.252.keytab.
W20260923 17:28:52.517436 14460 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260923 17:28:52.519047 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.252:42867
--webserver_interface=127.8.29.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.252.keytab
--principal=oryx/127.8.29.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:34369,127.8.29.253:41173,127.8.29.252:42867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:28:52.528429 14937 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:52.528945 14937 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:52.529117 14937 tablet_replica.cc:330] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: stopping tablet replica
I20260923 17:28:52.536581 14937 master.cc:552] Master@127.8.29.253:41173 shutdown complete.
I20260923 17:28:52.536758 14937 master_runner.cc:305] Clearing existing system tablet
I20260923 17:28:52.537226 14937 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:52.537395 14937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:52.546674 14937 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.008s sys 0.000s
I20260923 17:28:52.547982 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.547466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.0.0.1:43456)
0923 17:28:52.547660 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:52.547802 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:41173: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 17:28:52.548247 14460 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 -> Peer b894fbfff63a4a1297abadee53713db3 (127.8.29.253:41173): Couldn't send request to peer b894fbfff63a4a1297abadee53713db3. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.253:41173: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 17:28:52.550853 15041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:52.551231 14937 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:52.551534 14937 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:52.551622 14937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal
uuid: "b894fbfff63a4a1297abadee53713db3"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "42b00c6e55f25bbdc3c632d95aa46df0"
server_key_iv: "4fc9f317257316ba84630dd6efa0c968"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.556800 14937 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:52.563992 14937 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:28:52.564196 14937 log.cc:1196] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 17:28:52.564630 14937 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Deleting consensus metadata
I20260923 17:28:52.564910 14937 master_runner.cc:315] Copying system tablet from 127.8.29.254:34369
I20260923 17:28:52.567567 14937 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: tablet copy: Beginning tablet copy session from remote peer at address 127.8.29.254:34369
I20260923 17:28:52.589290 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.568393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49204 (local address 127.8.29.254:34369)
0923 17:28:52.568593 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.568601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.569021 (+ 420us) server_negotiation.cc:407] Connection header received
0923 17:28:52.569234 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.569239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.569323 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.569434 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:52.570524 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.571692 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.573531 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.574184 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.574316 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.581861 (+ 7545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:52.581870 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:52.581876 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:52.581930 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:52.585903 (+ 3973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.586940 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.586946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.586949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.587123 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:52.587612 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:52.587618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:52.587621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:52.588002 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:52.588166 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.588625 (+ 459us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.588862 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:28:52.589373 15047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.568369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:49204)
0923 17:28:52.568904 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:52.568923 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:52.569075 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:52.569479 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:52.569484 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:52.569994 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:52.570323 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.570337 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.571848 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.571854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.573359 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.573375 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.574351 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.574357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.574581 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.580268 (+ 5687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:52.580302 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:52.581490 (+ 1188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:52.586129 (+ 4639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.586136 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.586141 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.586795 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.587307 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:52.587312 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:52.587316 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:52.587496 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:52.588143 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:52.588148 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:52.588338 (+ 190us) client_negotiation.cc:769] Sending connection context
0923 17:28:52.588507 (+ 169us) client_negotiation.cc:240] Negotiation successful
0923 17:28:52.588872 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":254,"threads_started":1}
I20260923 17:28:52.590708 14496 tablet_copy_service.cc:140] P 6893904a76694629b825a55109145c87: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b894fbfff63a4a1297abadee53713db3 ({username='oryx', principal='oryx/127.8.29.253@KRBTEST.COM'} at 127.0.0.1:49204)
I20260923 17:28:52.590967 14496 tablet_copy_service.cc:161] P 6893904a76694629b825a55109145c87: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b894fbfff63a4a1297abadee53713db3 at {username='oryx', principal='oryx/127.8.29.253@KRBTEST.COM'} at 127.0.0.1:49204: session id = b894fbfff63a4a1297abadee53713db3-00000000000000000000000000000000
I20260923 17:28:52.595484 14496 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:28:52.597916 14937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:28:52.604889 14937 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: tablet copy: Starting download of 0 data blocks...
I20260923 17:28:52.605221 14937 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: tablet copy: Starting download of 1 WAL segments...
I20260923 17:28:52.608215 14937 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:28:52.613031 14496 tablet_copy_service.cc:342] P 6893904a76694629b825a55109145c87: Request end of tablet copy session b894fbfff63a4a1297abadee53713db3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.29.253@KRBTEST.COM'} at 127.0.0.1:49204
I20260923 17:28:52.613241 14496 tablet_copy_service.cc:433] P 6893904a76694629b825a55109145c87: ending tablet copy session b894fbfff63a4a1297abadee53713db3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b894fbfff63a4a1297abadee53713db3
I20260923 17:28:52.625034 14937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:52.628049 15048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:52.632056 15049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:52.642050 15051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:52.643785 14937 server_base.cc:1034] running on GCE node
I20260923 17:28:52.644143 14937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:52.644805 14937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:52.646009 14937 hybrid_clock.cc:648] HybridClock initialized: now 1790184532645948 us; error 38 us; skew 500 ppm
I20260923 17:28:52.646237 14937 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:52 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:52.648383 14937 init.cc:377] Logged in from keytab as oryx/127.8.29.253@KRBTEST.COM (short username oryx)
I20260923 17:28:52.649644 14937 webserver.cc:460] Webserver started at http://127.8.29.253:44787/ using document root <none> and password file <none>
I20260923 17:28:52.649991 14937 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:52.650087 14937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:52.653144 14937 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260923 17:28:52.655045 15058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:52.655454 14937 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260923 17:28:52.655573 14937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal
uuid: "b894fbfff63a4a1297abadee53713db3"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "42b00c6e55f25bbdc3c632d95aa46df0"
server_key_iv: "4fc9f317257316ba84630dd6efa0c968"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.655733 14937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:52.673183 14937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:52.677104 14937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:52.708542 14937 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.253:41173
I20260923 17:28:52.708595 15109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.253:41173 every 8 connection(s)
I20260923 17:28:52.709554 14937 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
I20260923 17:28:52.715034 15110 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:28:52.719129 15110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Bootstrap starting.
I20260923 17:28:52.738722 15110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:28:52.739382 15110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Bootstrap complete.
I20260923 17:28:52.740643 15110 raft_consensus.cc:348] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } }
I20260923 17:28:52.740967 15110 raft_consensus.cc:729] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b894fbfff63a4a1297abadee53713db3, State: Initialized, Role: LEARNER
I20260923 17:28:52.741225 15110 consensus_queue.cc:260] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } }
I20260923 17:28:52.742105 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } } }
I20260923 17:28:52.742517 15110 sys_catalog.cc:564] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:52.742635 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:28:52.751852 15124 catalog_manager.cc:1259] Loaded cluster ID: d7687137e2964f55b9c555ae17474c0a
I20260923 17:28:52.752095 15124 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: loading cluster ID for follower catalog manager: success
I20260923 17:28:52.755919 15124 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: acquiring CA information for follower catalog manager: success
I20260923 17:28:52.758697 15124 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:28:52.879225 14932 heartbeater.cc:503] Master 127.8.29.254:34369 was elected leader, sending a full tablet report...
W20260923 17:28:52.917541 15038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:52.917861 15038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:52.917989 15038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:52.926893 15038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:52.927047 15038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:52.927138 15038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:52.927225 15038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:52.927289 15038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:52.928157 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.920905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.0.0.1:43460)
0923 17:28:52.921105 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:52.921125 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:52.921258 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:52.922137 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:52.922141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:52.922165 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:52.922317 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.922324 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.924154 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.924158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.925540 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:52.925561 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.927310 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.927317 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:52.927557 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.927636 (+ 79us) client_negotiation.cc:769] Sending connection context
0923 17:28:52.927758 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 17:28:52.927895 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:28:52.928280 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:52.921163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43460 (local address 127.8.29.253:41173)
0923 17:28:52.921756 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:52.921762 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:52.921786 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:28:52.921857 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:52.921862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:52.921949 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:52.922088 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:52.922451 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.923978 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.925703 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:52.927039 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:52.927221 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:52.927487 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:52.927817 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 17:28:52.927959 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":304,"threads_started":1}
I20260923 17:28:52.929579 15085 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260923 17:28:52.930260 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6893904a76694629b825a55109145c87. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } } }
I20260923 17:28:52.930567 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:28:52.938042 15038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:34369,127.8.29.253:41173,127.8.29.252:42867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.252.keytab
--principal=oryx/127.8.29.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.252:42867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.29.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:52.939468 15038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:52.940816 15038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:52.953135 15130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:52.953104 15131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:52.953181 15133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:52.954321 15038 server_base.cc:1034] running on GCE node
I20260923 17:28:52.955178 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:52.956720 15038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:52.957943 15038 hybrid_clock.cc:648] HybridClock initialized: now 1790184532957890 us; error 50 us; skew 500 ppm
I20260923 17:28:52.958312 15038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:52 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:52.961388 15038 init.cc:377] Logged in from keytab as oryx/127.8.29.252@KRBTEST.COM (short username oryx)
I20260923 17:28:52.962922 15038 webserver.cc:460] Webserver started at http://127.8.29.252:39247/ using document root <none> and password file <none>
I20260923 17:28:52.963619 15038 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:52.963723 15038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:52.964027 15038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:52.967346 15038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data/instance:
uuid: "c001745d65fc469b9b595186ec0be467"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "109d53a2fab25a438a62d2e076a8697f"
server_key_iv: "bb9c1e2ff1ad57bda861a7956fb17a78"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.968122 15038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal/instance:
uuid: "c001745d65fc469b9b595186ec0be467"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "109d53a2fab25a438a62d2e076a8697f"
server_key_iv: "bb9c1e2ff1ad57bda861a7956fb17a78"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.973667 15038 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:28:52.977164 15140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:52.978665 15038 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:28:52.978833 15038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal
uuid: "c001745d65fc469b9b595186ec0be467"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "109d53a2fab25a438a62d2e076a8697f"
server_key_iv: "bb9c1e2ff1ad57bda861a7956fb17a78"
server_key_version: "encryptionkey@0"
I20260923 17:28:52.979019 15038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:53.008069 15038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:53.012241 15038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:53.012475 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:53.030144 15038 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.252:42867
I20260923 17:28:53.030161 15191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.252:42867 every 8 connection(s)
I20260923 17:28:53.031525 15038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
I20260923 17:28:53.032758 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 15038
I20260923 17:28:53.036090 15192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:53.043273 15192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.252@KRBTEST.COM
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.252@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:28:53.056264 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.034955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44744 (local address 127.8.29.252:42867)
0923 17:28:53.035705 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.035718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.035756 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:28:53.037423 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.037447 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.037872 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.038225 (+ 353us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.039805 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.041319 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.042884 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.043716 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.043932 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.048718 (+ 4786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.048726 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.048736 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.048770 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.051870 (+ 3100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.052780 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.052788 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.052799 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.052997 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.053347 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.053353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.053356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.053996 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.054209 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.054778 (+ 569us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.055175 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":312,"threads_started":1}
I20260923 17:28:53.060978 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.045898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:49210)
0923 17:28:53.046640 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.046693 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.046850 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.047365 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.047373 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.048145 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.048424 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.048434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.049427 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.049433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.050787 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.050812 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.051646 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.051654 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.051851 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.052500 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.052517 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.055253 (+ 2736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.058526 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.058534 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.058539 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.058990 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.059463 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.059466 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.059469 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.059601 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.060182 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.060187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.060316 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.060459 (+ 143us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.060700 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":342,"threads_started":1}
I20260923 17:28:53.060968 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.046044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49210 (local address 127.8.29.254:34369)
0923 17:28:53.046263 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.046270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.046791 (+ 521us) server_negotiation.cc:407] Connection header received
0923 17:28:53.047031 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.047044 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.047138 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.047267 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.048576 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.049280 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.050959 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.051382 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.051521 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.055433 (+ 3912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.055439 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.055442 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.055477 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.058369 (+ 2892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.059134 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.059139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.059142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.059360 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.059734 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.059738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.059740 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.060070 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.060193 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.060575 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.060747 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260923 17:28:53.062238 15192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.252@KRBTEST.COM for oryx/127.8.29.253@KRBTEST.COM
I20260923 17:28:53.074226 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.062984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43474 (local address 127.8.29.253:41173)
0923 17:28:53.063167 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.063173 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.063191 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:53.063237 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.063241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.063296 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.063374 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.064013 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.064727 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.065879 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.066302 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.066408 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.069366 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.069371 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.069374 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.069404 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.071829 (+ 2425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.072419 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.072423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.072425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.072560 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.072896 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.072899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.072901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.073171 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.073296 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.073690 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.073903 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:28:53.074215 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.062796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.0.0.1:43474)
0923 17:28:53.062967 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.062982 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.063078 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.063407 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.063411 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.063744 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.063879 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.063886 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.064846 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.064851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.065751 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.065762 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.066445 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.066451 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.066662 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.067212 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.067234 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.069183 (+ 1949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.071947 (+ 2764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.071953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.071957 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.072314 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.072676 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.072680 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.072683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.072801 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.073277 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.073281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.073423 (+ 142us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.073553 (+ 130us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.073842 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 17:28:53.075291 15192 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184532, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.252@KRBTEST.COM for oryx/127.8.29.252@KRBTEST.COM
I20260923 17:28:53.086959 15196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.075887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:42867 (local address 127.0.0.1:44756)
0923 17:28:53.076079 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.076098 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.076186 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.076510 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.076513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.076843 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.076982 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.076990 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.077862 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.077865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.078769 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.078781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.079488 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.079492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.079662 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.080120 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.080135 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.081931 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.084603 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.084608 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.084611 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.084958 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.085390 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.085395 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.085398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.085533 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.086148 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.086151 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.086294 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.086400 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.086650 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 17:28:53.087110 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.076054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44756 (local address 127.8.29.252:42867)
0923 17:28:53.076186 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.076190 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.076205 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:28:53.076303 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.076308 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.076378 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.076446 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.077092 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.077756 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.078888 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.079287 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.079418 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.082075 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.082081 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.082085 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.082118 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.084474 (+ 2356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.085126 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.085131 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.085133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.085269 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.085645 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.085648 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.085651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.086040 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.086158 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.086540 (+ 382us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.086790 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":69,"server-negotiator.queue_time_us":48}
I20260923 17:28:53.091851 15192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: Bootstrap starting.
I20260923 17:28:53.094996 15192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:53.096019 15192 log.cc:824] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:53.098678 15192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: No bootstrap required, opened a new log
I20260923 17:28:53.106366 15192 raft_consensus.cc:348] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } }
I20260923 17:28:53.106829 15192 raft_consensus.cc:374] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:53.106923 15192 raft_consensus.cc:729] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c001745d65fc469b9b595186ec0be467, State: Initialized, Role: FOLLOWER
I20260923 17:28:53.107550 15192 consensus_queue.cc:260] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } }
I20260923 17:28:53.108461 15198 sys_catalog.cc:455] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } } }
I20260923 17:28:53.108767 15198 sys_catalog.cc:458] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:28:53.109465 15192 sys_catalog.cc:564] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:28:53.117724 15209 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:28:53.117851 15209 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:28:53.122061 15038 master_runner.cc:428] Detected that this master c001745d65fc469b9b595186ec0be467 is joining an existing cluster
I20260923 17:28:53.122174 15038 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.29.252:42867
I20260923 17:28:53.142122 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.126034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49226 (local address 127.8.29.254:34369)
0923 17:28:53.126203 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.126207 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.126871 (+ 664us) server_negotiation.cc:407] Connection header received
0923 17:28:53.127036 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.127041 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.127125 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.127216 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.128769 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.129880 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.131899 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.132563 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.132690 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.135366 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.135373 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.135378 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.135421 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.138748 (+ 3327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.139560 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.139578 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.139582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.139767 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.140272 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.140277 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.140281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.140630 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.140754 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.141459 (+ 705us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.141702 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:28:53.143357 15128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.127369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43480 (local address 127.8.29.253:41173)
0923 17:28:53.127536 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.127541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.128122 (+ 581us) server_negotiation.cc:407] Connection header received
0923 17:28:53.128355 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.128361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.128436 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.128504 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.129961 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.131029 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.133019 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.133620 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.133737 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.136448 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.136455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.136460 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.136501 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.139964 (+ 3463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.140711 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.140717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.140720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.140885 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.141332 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.141336 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.141339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.141666 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.141775 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.142773 (+ 998us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.143020 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:28:53.143381 15215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.126086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:49226)
0923 17:28:53.126760 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.126776 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.126912 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.127406 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.127410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.128157 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.128602 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.128615 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.130046 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.130050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.131740 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.131753 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.132984 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.132990 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.133165 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.133886 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.133909 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.135137 (+ 1228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.138907 (+ 3770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.138912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.138917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.139412 (+ 495us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.139940 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.139944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.139947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.140142 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.140828 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.140832 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.141132 (+ 300us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.142390 (+ 1258us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.142832 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":514,"spinlock_wait_cycles":23808,"thread_start_us":357,"threads_started":1}
I20260923 17:28:53.144656 15216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.126978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.0.0.1:43480)
0923 17:28:53.128021 (+ 1043us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.128039 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.128210 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.128686 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.128689 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.129455 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.129787 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.129801 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.131254 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.131259 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.132868 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.132880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.134018 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.134023 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.134212 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.134821 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.134844 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.136247 (+ 1403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.140134 (+ 3887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.140138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.140141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.140575 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.141024 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.141029 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.141033 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.141213 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.141872 (+ 659us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.141876 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.142137 (+ 261us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.143964 (+ 1827us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.144221 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"thread_start_us":728,"threads_started":1}
I20260923 17:28:53.153733 15217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.128032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:42867 (local address 127.0.0.1:44764)
0923 17:28:53.138440 (+ 10408us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.138484 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.138843 (+ 359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.139294 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.139299 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.139947 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.140319 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.140331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.142223 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.142229 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.144105 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.144120 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.145209 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.145216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.145404 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.145952 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.145983 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.147172 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.150980 (+ 3808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.150985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.150989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.151404 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.151788 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.151792 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.151795 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.151935 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.152615 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.152621 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.152779 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.152928 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.153225 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10237,"thread_start_us":10567,"threads_started":1}
I20260923 17:28:53.153780 15195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.128343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44764 (local address 127.8.29.252:42867)
0923 17:28:53.128527 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.128532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.138581 (+ 10049us) server_negotiation.cc:407] Connection header received
0923 17:28:53.138967 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.138973 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.139036 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.139110 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.140466 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.141619 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.144257 (+ 2638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.145021 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.145143 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.147391 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.147397 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.147402 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.147450 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.150861 (+ 3411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.151525 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.151530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.151533 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.151668 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.152068 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.152073 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.152076 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.152488 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.152677 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.153072 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.153393 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":62,"server-negotiator.queue_time_us":62}
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184530, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.254@KRBTEST.COM for oryx/127.8.29.252@KRBTEST.COM
I20260923 17:28:53.162381 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 14537
I20260923 17:28:53.164933 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.149100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:42867 (local address 127.0.0.1:44766)
0923 17:28:53.149367 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.149386 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.149494 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.151303 (+ 1809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.151309 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.151752 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.151943 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.151955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.153357 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.153363 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.154680 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.154696 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.155415 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.155421 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.155580 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.156128 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.156150 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.158406 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.161691 (+ 3285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.161696 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.161701 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.162161 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.162644 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.162650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.162653 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.162826 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.163870 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.163875 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.164062 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.164194 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.164493 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20260923 17:28:53.166098 15218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.149199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44766 (local address 127.8.29.252:42867)
0923 17:28:53.150916 (+ 1717us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.150922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.150948 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:28:53.151046 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.151055 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.151138 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.151248 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.152107 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.153206 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.154840 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.155257 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.155813 (+ 556us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.158635 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.158640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.158644 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.158670 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.161563 (+ 2893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.162291 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.162297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.162300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.162447 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.162948 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.162953 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.162956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.163315 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.165328 (+ 2013us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.165457 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.165701 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1585,"thread_start_us":1436,"threads_started":1}
I20260923 17:28:53.167600 14476 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.29.252:42867: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "c001745d65fc469b9b595186ec0be467"
member_type: NON_VOTER
last_known_addr {
host: "127.8.29.252"
port: 42867
}
attrs {
promote: true
}
}
dest_uuid: "6893904a76694629b825a55109145c87"
cas_config_opid_index: 5
I20260923 17:28:53.168511 14476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } }
I20260923 17:28:53.172079 15085 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 LEARNER]: Refusing update from remote peer 6893904a76694629b825a55109145c87: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:28:53.172392 15167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:53.170674 15035 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c001745d65fc469b9b595186ec0be467 (127.8.29.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.176041 15035 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.29.252:42867
I20260923 17:28:53.176579 15167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 FOLLOWER]: Refusing update from remote peer 6893904a76694629b825a55109145c87: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 17:28:53.176937 15038 master.cc:537] Master@127.8.29.252:42867 shutting down...
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.193@KRBTEST.COM
I20260923 17:28:53.187628 15221 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 17:28:53.190704 15222 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:53.195770 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.196126 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:53.196842 15219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.163637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60320 (local address 127.8.29.193:35875)
0923 17:28:53.164209 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.164216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.164241 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:28:53.164295 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.164301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.164506 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.164670 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.165947 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.167245 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.169144 (+ 1899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.170180 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.171028 (+ 848us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.187307 (+ 16279us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.187330 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.187353 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.187414 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.191445 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.192395 (+ 950us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.192404 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.192408 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.192564 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.193134 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.193139 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.193142 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.193580 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.195021 (+ 1441us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.195202 (+ 181us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.195471 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":276,"threads_started":1}
I20260923 17:28:53.198371 14660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:53.198642 15085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER c001745d65fc469b9b595186ec0be467 (127.8.29.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.204309 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: NON_VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: true } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.204627 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:28:53.205705 15221 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: attempting to promote NON_VOTER b894fbfff63a4a1297abadee53713db3 to VOTER
I20260923 17:28:53.206378 15221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } }
I20260923 17:28:53.208282 15085 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 LEARNER]: Refusing update from remote peer 6893904a76694629b825a55109145c87: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 17:28:53.208778 15220 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260923 17:28:53.211175 15221 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b894fbfff63a4a1297abadee53713db3 (127.8.29.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.214864 15221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.215106 15221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:53.215552 15220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.215797 15220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:53.220819 15038 raft_consensus.cc:2229] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:53.221268 15038 raft_consensus.cc:2258] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:53.221465 15038 tablet_replica.cc:330] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: stopping tablet replica
I20260923 17:28:53.225534 15085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, b894fbfff63a4a1297abadee53713db3 (127.8.29.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.230343 15038 master.cc:552] Master@127.8.29.252:42867 shutdown complete.
I20260923 17:28:53.230436 15038 master_runner.cc:305] Clearing existing system tablet
I20260923 17:28:53.230854 15038 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:53.230942 15038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:53.232187 15113 sys_catalog.cc:455] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.232466 15113 sys_catalog.cc:458] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:28:53.234719 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.234181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:42867 (local address 127.0.0.1:44782)
0923 17:28:53.234376 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.234508 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:42867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260923 17:28:53.235069 14460 consensus_peers.cc:487] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 -> Peer c001745d65fc469b9b595186ec0be467 (127.8.29.252:42867): Couldn't send request to peer c001745d65fc469b9b595186ec0be467. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.252:42867: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 17:28:53.236752 15038 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260923 17:28:53.238178 15228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:53.238517 15038 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:53.238899 15038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 17:28:53.239009 15038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal
uuid: "c001745d65fc469b9b595186ec0be467"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "109d53a2fab25a438a62d2e076a8697f"
server_key_iv: "bb9c1e2ff1ad57bda861a7956fb17a78"
server_key_version: "encryptionkey@0"
I20260923 17:28:53.242532 15038 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:53.247967 15038 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 17:28:53.248126 15038 log.cc:1196] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 17:28:53.248566 15038 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: Deleting consensus metadata
I20260923 17:28:53.248804 15038 master_runner.cc:315] Copying system tablet from 127.8.29.254:34369
I20260923 17:28:53.265926 15038 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: tablet copy: Beginning tablet copy session from remote peer at address 127.8.29.254:34369
I20260923 17:28:53.285358 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.266841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:49234)
0923 17:28:53.270926 (+ 4085us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.270947 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.271126 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.271667 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.271672 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.272217 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.272510 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.272523 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.273975 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.273980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.275557 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.275573 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.276532 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.276538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.276744 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.277429 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.277453 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.278526 (+ 1073us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.282367 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.282373 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.282377 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.282919 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.283364 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.283370 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.283375 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.283551 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.284249 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.284255 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.284442 (+ 187us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.284579 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.284922 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3937,"thread_start_us":3787,"threads_started":1}
I20260923 17:28:53.285363 15008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.266853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49234 (local address 127.8.29.254:34369)
0923 17:28:53.267075 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.267082 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.271088 (+ 4006us) server_negotiation.cc:407] Connection header received
0923 17:28:53.271249 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.271255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.271363 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.271472 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.272710 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.273834 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.275717 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.276373 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.276497 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.278741 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.278748 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.278753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.278792 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.282207 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.283057 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.283063 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.283067 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.283222 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.283686 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.283692 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.283695 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.284080 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.284244 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.284716 (+ 472us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.284966 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260923 17:28:53.286315 14496 tablet_copy_service.cc:140] P 6893904a76694629b825a55109145c87: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer c001745d65fc469b9b595186ec0be467 ({username='oryx', principal='oryx/127.8.29.252@KRBTEST.COM'} at 127.0.0.1:49234)
I20260923 17:28:53.286509 14496 tablet_copy_service.cc:161] P 6893904a76694629b825a55109145c87: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer c001745d65fc469b9b595186ec0be467 at {username='oryx', principal='oryx/127.8.29.252@KRBTEST.COM'} at 127.0.0.1:49234: session id = c001745d65fc469b9b595186ec0be467-00000000000000000000000000000000
I20260923 17:28:53.289973 14496 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 17:28:53.291980 15038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 17:28:53.299211 15038 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: tablet copy: Starting download of 0 data blocks...
I20260923 17:28:53.299501 15038 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: tablet copy: Starting download of 1 WAL segments...
I20260923 17:28:53.302114 15038 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 17:28:53.306878 14496 tablet_copy_service.cc:342] P 6893904a76694629b825a55109145c87: Request end of tablet copy session c001745d65fc469b9b595186ec0be467-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.29.252@KRBTEST.COM'} at 127.0.0.1:49234
I20260923 17:28:53.307034 14496 tablet_copy_service.cc:433] P 6893904a76694629b825a55109145c87: ending tablet copy session c001745d65fc469b9b595186ec0be467-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer c001745d65fc469b9b595186ec0be467
I20260923 17:28:53.318513 15038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:53.321127 15235 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:53.323920 15236 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:53.336812 15038 server_base.cc:1034] running on GCE node
W20260923 17:28:53.337419 15238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:53.338783 15038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:53.340921 15038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:53.342188 15038 hybrid_clock.cc:648] HybridClock initialized: now 1790184533342147 us; error 41 us; skew 500 ppm
I20260923 17:28:53.342553 15038 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184533, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:53.344769 15038 init.cc:377] Logged in from keytab as oryx/127.8.29.252@KRBTEST.COM (short username oryx)
I20260923 17:28:53.348212 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 14537
I20260923 17:28:53.348868 15038 webserver.cc:460] Webserver started at http://127.8.29.252:44911/ using document root <none> and password file <none>
I20260923 17:28:53.349220 15038 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:53.349309 15038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:53.352378 15038 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 17:28:53.354498 15245 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:53.354969 15038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.002s sys 0.000s
I20260923 17:28:53.355122 15038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal
uuid: "c001745d65fc469b9b595186ec0be467"
format_stamp: "Formatted at 2026-09-23 17:28:52 on dist-test-slave-p7jt"
server_key: "109d53a2fab25a438a62d2e076a8697f"
server_key_iv: "bb9c1e2ff1ad57bda861a7956fb17a78"
server_key_version: "encryptionkey@0"
I20260923 17:28:53.355319 15038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:53.364070 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:35875
--local_ip_for_outbound_sockets=127.8.29.193
--tserver_master_addrs=127.8.29.254:34369
--webserver_port=34339
--webserver_interface=127.8.29.193
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab
--principal=oryx/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.29.254:34369,127.8.29.253:41173
--tserver_master_addrs=127.8.29.254:34369,127.8.29.253:41173,127.8.29.252:42867 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:28:53.387460 15038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:53.391428 15038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:53.422487 15038 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.252:42867
I20260923 17:28:53.422539 15297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.252:42867 every 8 connection(s)
I20260923 17:28:53.423671 15038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
I20260923 17:28:53.428771 15298 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:28:53.432735 15298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: Bootstrap starting.
I20260923 17:28:53.452242 15298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:28:53.452827 15298 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: Bootstrap complete.
I20260923 17:28:53.453922 15298 raft_consensus.cc:348] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } }
I20260923 17:28:53.454219 15298 raft_consensus.cc:729] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c001745d65fc469b9b595186ec0be467, State: Initialized, Role: LEARNER
I20260923 17:28:53.454517 15298 consensus_queue.cc:260] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } }
I20260923 17:28:53.455458 15301 sys_catalog.cc:455] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.455892 15298 sys_catalog.cc:564] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:53.455901 15301 sys_catalog.cc:458] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: This master's current role is: LEARNER
I20260923 17:28:53.467609 15312 catalog_manager.cc:1259] Loaded cluster ID: d7687137e2964f55b9c555ae17474c0a
I20260923 17:28:53.467736 15312 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: loading cluster ID for follower catalog manager: success
I20260923 17:28:53.471352 15312 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: acquiring CA information for follower catalog manager: success
I20260923 17:28:53.473999 15312 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:28:53.611241 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.604963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44794 (local address 127.8.29.252:42867)
0923 17:28:53.605481 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.605485 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.605503 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:53.605563 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.605567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.605631 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.605723 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:53.606147 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.607163 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.609434 (+ 2271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.610271 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.610367 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.610593 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.610865 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.610957 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":285,"threads_started":1}
I20260923 17:28:53.611282 15031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.604777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:42867 (local address 127.0.0.1:44794)
0923 17:28:53.604974 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.604995 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.605127 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.605776 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.605782 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.605810 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:53.606001 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.606012 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.607320 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.607326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.609268 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.609286 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.610487 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.610494 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.610703 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.610741 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.610851 (+ 110us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.611001 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:28:53.612214 15273 raft_consensus.cc:1203] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 17:28:53.612819 15301 sys_catalog.cc:455] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6893904a76694629b825a55109145c87. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:53.613126 15301 sys_catalog.cc:458] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: This master's current role is: LEARNER
W20260923 17:28:53.721136 15250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:53.721403 15250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:53.721519 15250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:53.728926 15250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:53.729049 15250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:53.729193 15250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:28:53.739533 15250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab
--principal=oryx/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:35875
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=34339
--tserver_master_addrs=127.8.29.254:34369,127.8.29.253:41173,127.8.29.252:42867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:53.740854 15250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:53.742208 15250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:53.755812 15318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:53.755812 15321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:53.755862 15319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:53.757061 15250 server_base.cc:1034] running on GCE node
I20260923 17:28:53.757709 15250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:53.758695 15250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:53.759939 15250 hybrid_clock.cc:648] HybridClock initialized: now 1790184533759854 us; error 67 us; skew 500 ppm
I20260923 17:28:53.760296 15250 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184533, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:53.763490 15250 init.cc:377] Logged in from keytab as oryx/127.8.29.193@KRBTEST.COM (short username oryx)
I20260923 17:28:53.765054 15250 webserver.cc:460] Webserver started at http://127.8.29.193:34339/ using document root <none> and password file <none>
I20260923 17:28:53.765806 15250 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:53.765936 15250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:53.771672 15250 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:28:53.774998 15328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:53.776674 15250 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:28:53.776842 15250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "455cb603b43346f0b1d56e6dc06f589b"
format_stamp: "Formatted at 2026-09-23 17:28:50 on dist-test-slave-p7jt"
server_key: "63ca067b67201608d61476c61bda2adb"
server_key_iv: "ebbda28ad054523e059b4eb1b788ca60"
server_key_version: "encryptionkey@0"
I20260923 17:28:53.777900 15250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:53.791594 15250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:53.795846 15250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:53.796162 15250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:53.797144 15250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:53.798578 15250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:53.798677 15250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:53.798779 15250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:53.798954 15250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:53.829064 15250 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:35875
I20260923 17:28:53.829126 15440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:35875 every 8 connection(s)
I20260923 17:28:53.830446 15250 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:28:53.837561 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 15250
I20260923 17:28:53.837849 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 14671
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184533, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.253@KRBTEST.COM
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184533, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184533, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.252@KRBTEST.COM
I20260923 17:28:53.862888 15313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.835970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:54901 (local address 127.8.29.252:42867)
0923 17:28:53.836131 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.836136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.836198 (+ 62us) server_negotiation.cc:407] Connection header received
0923 17:28:53.838664 (+ 2466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.838670 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.838752 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.838827 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.843195 (+ 4368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.844441 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.849621 (+ 5180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.850312 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.850414 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.854891 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.854900 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.854905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.854952 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.858339 (+ 3387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.859553 (+ 1214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.859560 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.859565 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.859759 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.860424 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.860430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.860434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.860852 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.861007 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.862182 (+ 1175us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.862489 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:28:53.863898 15445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.833320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:48815 (local address 127.8.29.254:34369)
0923 17:28:53.833720 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.833724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.834727 (+ 1003us) server_negotiation.cc:407] Connection header received
0923 17:28:53.838341 (+ 3614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.838346 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.838436 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.840499 (+ 2063us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.840705 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.841878 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.848572 (+ 6694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.849237 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.849363 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.855135 (+ 5772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.855143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.855148 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.855195 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.859564 (+ 4369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.860458 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.860464 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.860468 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.860622 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.861061 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.861067 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.861070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.861440 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.861579 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.863243 (+ 1664us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.863502 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":199,"threads_started":1}
I20260923 17:28:53.865320 15446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.833778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.193:48815)
0923 17:28:53.834563 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.834616 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.837187 (+ 2571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.839345 (+ 2158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.839349 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.839826 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.840305 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.840332 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.846551 (+ 6219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.846557 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.848245 (+ 1688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.848266 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.849427 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.849433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.849612 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.850196 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.850217 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.854839 (+ 4622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.859757 (+ 4918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.859762 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.859767 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.860303 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.860749 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.860755 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.860759 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.860938 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.861651 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.861656 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.861829 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.862048 (+ 219us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.862692 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":297,"threads_started":1}
I20260923 17:28:53.867087 15442 heartbeater.cc:347] Connected to a master server at 127.8.29.254:34369
I20260923 17:28:53.867446 15442 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:53.868462 15442 heartbeater.cc:511] Master 127.8.29.254:34369 requested a full tablet report, sending...
I20260923 17:28:53.869467 15449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.835607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:42867 (local address 127.8.29.193:54901)
0923 17:28:53.836097 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.836130 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.838189 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.839043 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.839048 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.842602 (+ 3554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.842996 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.843011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.844696 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.844702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.846303 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.846318 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.850761 (+ 4443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.850771 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.851033 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.851637 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.851660 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.854564 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.858566 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.858578 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.858604 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.859388 (+ 784us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.859908 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.859913 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.859918 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.860289 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.861109 (+ 820us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.861119 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.861632 (+ 513us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.863093 (+ 1461us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.863414 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"spinlock_wait_cycles":28416,"thread_start_us":524,"threads_started":1}
I20260923 17:28:53.870864 14476 ts_manager.cc:194] Re-registered known tserver with Master: 455cb603b43346f0b1d56e6dc06f589b (127.8.29.193:35875)
I20260923 17:28:53.871462 15447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.834984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.8.29.193:34361)
0923 17:28:53.835580 (+ 596us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:53.835601 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:53.837198 (+ 1597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:53.838337 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:53.838346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:53.839341 (+ 995us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:53.840318 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.840333 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.841997 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.842004 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.844619 (+ 2615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:53.844635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.845818 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.845826 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:53.846194 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.847142 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:53.847178 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:53.864294 (+ 17116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:53.868732 (+ 4438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.868736 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.868738 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.869127 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.869561 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:53.869566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:53.869569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:53.869747 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:53.870450 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:53.870487 (+ 37us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:53.870649 (+ 162us) client_negotiation.cc:769] Sending connection context
0923 17:28:53.870790 (+ 141us) client_negotiation.cc:240] Negotiation successful
0923 17:28:53.871072 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"mutex_wait_us":1,"thread_start_us":266,"threads_started":1}
I20260923 17:28:53.871920 14476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:48815
I20260923 17:28:53.872311 15448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.835207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:34361 (local address 127.8.29.253:41173)
0923 17:28:53.837820 (+ 2613us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.837824 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.837845 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:28:53.837894 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.837897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.837977 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.838063 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.840540 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.841799 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.844852 (+ 3053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.845548 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.848720 (+ 3172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.865219 (+ 16499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.865225 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.865229 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.865264 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.868424 (+ 3160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.869270 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.869276 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.869280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.869426 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.869882 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.869888 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.869890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.870242 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.870387 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.871750 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.871978 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2517,"thread_start_us":2409,"threads_started":1}
I20260923 17:28:53.872566 15441 heartbeater.cc:347] Connected to a master server at 127.8.29.252:42867
I20260923 17:28:53.872716 15441 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:53.873196 15441 heartbeater.cc:511] Master 127.8.29.252:42867 requested a full tablet report, sending...
I20260923 17:28:53.873941 15443 heartbeater.cc:347] Connected to a master server at 127.8.29.253:41173
I20260923 17:28:53.874063 15443 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:53.875272 15263 ts_manager.cc:194] Registered new tserver with Master: 455cb603b43346f0b1d56e6dc06f589b (127.8.29.193:35875)
I20260923 17:28:53.876255 15443 heartbeater.cc:511] Master 127.8.29.253:41173 requested a full tablet report, sending...
I20260923 17:28:53.877837 15075 ts_manager.cc:194] Registered new tserver with Master: 455cb603b43346f0b1d56e6dc06f589b (127.8.29.193:35875)
Sep 23 17:28:53 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.194@KRBTEST.COM
I20260923 17:28:53.884784 15451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:53.838880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41558 (local address 127.8.29.194:34613)
0923 17:28:53.866620 (+ 27740us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:53.866627 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:53.866652 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:28:53.866708 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:53.866714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:53.866927 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:53.867062 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:53.869693 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.870960 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.872490 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:53.873336 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:53.873475 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:53.879094 (+ 5619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:53.879108 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:53.879123 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:53.879164 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:53.882065 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.882881 (+ 816us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.882888 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.882890 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.883022 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:53.883408 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:53.883410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:53.883412 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:53.883748 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:53.883973 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:53.884232 (+ 259us) server_negotiation.cc:299] Negotiation successful
0923 17:28:53.884427 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27541,"thread_start_us":27391,"threads_started":1}
I20260923 17:28:53.886147 14794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:53.972935 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 14671
I20260923 17:28:53.986933 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:34613
--local_ip_for_outbound_sockets=127.8.29.194
--tserver_master_addrs=127.8.29.254:34369
--webserver_port=46555
--webserver_interface=127.8.29.194
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab
--principal=oryx/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.29.254:34369,127.8.29.253:41173
--tserver_master_addrs=127.8.29.254:34369,127.8.29.253:41173,127.8.29.252:42867 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:28:54.117590 15035 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: NON_VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: true } } }
I20260923 17:28:54.117933 15035 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:54.119582 15454 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: attempting to promote NON_VOTER c001745d65fc469b9b595186ec0be467 to VOTER
I20260923 17:28:54.120167 15454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } }
I20260923 17:28:54.121966 15085 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Refusing update from remote peer 6893904a76694629b825a55109145c87: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 17:28:54.122080 15273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 LEARNER]: Refusing update from remote peer 6893904a76694629b825a55109145c87: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260923 17:28:54.122706 15455 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 17:28:54.123171 15035 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260923 17:28:54.125548 15455 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c001745d65fc469b9b595186ec0be467 (127.8.29.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:54.126196 15085 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c001745d65fc469b9b595186ec0be467 (127.8.29.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:54.134563 15454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:54.135052 15454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:54.135444 15455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:54.135764 15455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:54.136373 15458 sys_catalog.cc:455] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:54.136770 15458 sys_catalog.cc:458] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:28:54.138687 15454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:54.138944 15454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:54.139367 15273 raft_consensus.cc:2932] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, c001745d65fc469b9b595186ec0be467 (127.8.29.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:54.144550 15459 sys_catalog.cc:455] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6893904a76694629b825a55109145c87" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:54.144867 15459 sys_catalog.cc:458] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 17:28:54.339411 15453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:54.339699 15453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:54.339794 15453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:54.347296 15453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:54.347424 15453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:54.347564 15453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:28:54.357777 15453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab
--principal=oryx/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:34613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=46555
--tserver_master_addrs=127.8.29.254:34369,127.8.29.253:41173,127.8.29.252:42867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:54.359277 15453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:54.360628 15453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:54.373809 15464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:54.373849 15465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:54.373769 15467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:54.374948 15453 server_base.cc:1034] running on GCE node
I20260923 17:28:54.375561 15453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:54.376519 15453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:54.377725 15453 hybrid_clock.cc:648] HybridClock initialized: now 1790184534377675 us; error 44 us; skew 500 ppm
I20260923 17:28:54.378037 15453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:54 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184534, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:54.381186 15453 init.cc:377] Logged in from keytab as oryx/127.8.29.194@KRBTEST.COM (short username oryx)
I20260923 17:28:54.382607 15453 webserver.cc:460] Webserver started at http://127.8.29.194:46555/ using document root <none> and password file <none>
I20260923 17:28:54.383328 15453 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:54.383461 15453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:54.389228 15453 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:28:54.392066 15474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:54.393500 15453 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260923 17:28:54.393663 15453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "d7c0e000eae64596a8d6eb329c4598b9"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-p7jt"
server_key: "015ca976908037e18eaeffefd22d5a19"
server_key_iv: "44465d0ccc60948360057cd1ab065da4"
server_key_version: "encryptionkey@0"
I20260923 17:28:54.394675 15453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:54.427042 15453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:54.431623 15453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:54.431900 15453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:54.432902 15453 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:54.434304 15453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:54.434401 15453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:54.434531 15453 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:54.434600 15453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:54.465025 15453 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:34613
I20260923 17:28:54.465104 15586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:34613 every 8 connection(s)
I20260923 17:28:54.466566 15453 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:28:54.471318 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 15453
I20260923 17:28:54.471604 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 14804
Sep 23 17:28:54 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184534, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
Sep 23 17:28:54 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.195@KRBTEST.COM
Sep 23 17:28:54 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184534, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.253@KRBTEST.COM
I20260923 17:28:54.496762 15591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:54.469531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:44035 (local address 127.8.29.254:34369)
0923 17:28:54.470015 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:54.470019 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:54.471144 (+ 1125us) server_negotiation.cc:407] Connection header received
0923 17:28:54.473720 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:54.473726 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:54.473831 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:54.473912 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:54.476443 (+ 2531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.477679 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.480442 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.481136 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.481252 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:54.486630 (+ 5378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:54.486638 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:54.486644 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:54.486704 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:54.490431 (+ 3727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.491648 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.491655 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.491659 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.491845 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.492405 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.492411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.492414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.492816 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:54.492975 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:54.496115 (+ 3140us) server_negotiation.cc:299] Negotiation successful
0923 17:28:54.496377 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":237,"threads_started":1}
I20260923 17:28:54.499508 15592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:54.470128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.194:44035)
0923 17:28:54.470890 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:54.471009 (+ 119us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:54.473371 (+ 2362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:54.474545 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:54.474558 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:54.475353 (+ 795us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:54.476214 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:54.476238 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.477911 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.477916 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:54.480244 (+ 2328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:54.480262 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.481697 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.481704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:54.482045 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:54.482967 (+ 922us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:54.482998 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:54.486323 (+ 3325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:54.490680 (+ 4357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:54.490691 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:54.490716 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:54.491436 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:54.492005 (+ 569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:54.492010 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:54.492015 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:54.492279 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:54.493122 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:54.493132 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:54.495592 (+ 2460us) client_negotiation.cc:769] Sending connection context
0923 17:28:54.497062 (+ 1470us) client_negotiation.cc:240] Negotiation successful
0923 17:28:54.497921 (+ 859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"mutex_wait_us":5,"thread_start_us":333,"threads_started":1}
I20260923 17:28:54.501407 15588 heartbeater.cc:347] Connected to a master server at 127.8.29.254:34369
I20260923 17:28:54.501717 15588 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:54.502843 15588 heartbeater.cc:511] Master 127.8.29.254:34369 requested a full tablet report, sending...
Sep 23 17:28:54 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184534, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.252@KRBTEST.COM
I20260923 17:28:54.504503 15593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:54.471449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.8.29.194:38453)
0923 17:28:54.481555 (+ 10106us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:54.481587 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:54.481728 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:54.487865 (+ 6137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:54.487870 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:54.488501 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:54.488805 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:54.488817 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.490320 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.490326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:54.491906 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:54.491921 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.492967 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.492974 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:54.493170 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:54.493835 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:54.493855 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:54.497196 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:54.501137 (+ 3941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:54.501142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:54.501146 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:54.502045 (+ 899us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:54.502525 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:54.502529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:54.502532 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:54.502684 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:54.503437 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:54.503442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:54.503605 (+ 163us) client_negotiation.cc:769] Sending connection context
0923 17:28:54.503753 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 17:28:54.504064 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9933,"mutex_wait_us":45,"thread_start_us":11461,"threads_started":1}
I20260923 17:28:54.504912 14476 ts_manager.cc:194] Re-registered known tserver with Master: d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613)
I20260923 17:28:54.505175 15594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:54.471576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:38453 (local address 127.8.29.253:41173)
0923 17:28:54.487436 (+ 15860us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:54.487442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:54.487460 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:54.487514 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:54.487519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:54.487600 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:54.488954 (+ 1354us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:54.489148 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.490163 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.492116 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.492753 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.493243 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:54.497417 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:54.497425 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:54.497430 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:54.497486 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:54.500963 (+ 3477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.502205 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.502211 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.502214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.502363 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.502827 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.502833 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.502837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.503216 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:54.503375 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:54.504693 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0923 17:28:54.504910 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15743,"thread_start_us":16118,"threads_started":1}
I20260923 17:28:54.506029 14476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:44035
I20260923 17:28:54.506070 15596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:54.472866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40560 (local address 127.8.29.195:44201)
0923 17:28:54.483035 (+ 10169us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:54.483045 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:54.483077 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:28:54.483148 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:54.483157 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:54.483455 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:54.483621 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:54.486027 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.487634 (+ 1607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.489422 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.490293 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.490399 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:54.496755 (+ 6356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:54.496775 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:54.496795 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:54.496849 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:54.501584 (+ 4735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.502647 (+ 1063us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.502659 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.502663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.502833 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.503939 (+ 1106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.503956 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.503959 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.504390 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:54.504697 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:54.505376 (+ 679us) server_negotiation.cc:299] Negotiation successful
0923 17:28:54.505660 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9971,"thread_start_us":9800,"threads_started":1}
I20260923 17:28:54.506408 15589 heartbeater.cc:347] Connected to a master server at 127.8.29.253:41173
I20260923 17:28:54.506601 15589 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:54.507014 15589 heartbeater.cc:511] Master 127.8.29.253:41173 requested a full tablet report, sending...
I20260923 17:28:54.507665 14927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:54.508641 15075 ts_manager.cc:194] Registered new tserver with Master: d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613)
I20260923 17:28:54.514344 15598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:54.487008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:42867 (local address 127.8.29.194:55029)
0923 17:28:54.493922 (+ 6914us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:54.493944 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:54.494100 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:54.494523 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:54.494528 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:54.495788 (+ 1260us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:54.496115 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:54.496129 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.497689 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.497695 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:54.499164 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:54.499177 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.500775 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.500784 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:54.500960 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:54.501560 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:54.501581 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:54.506195 (+ 4614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:54.511110 (+ 4915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:54.511117 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:54.511122 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:54.511658 (+ 536us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:54.512257 (+ 599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:54.512263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:54.512268 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:54.512603 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:54.513169 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:54.513175 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:54.513425 (+ 250us) client_negotiation.cc:769] Sending connection context
0923 17:28:54.513595 (+ 170us) client_negotiation.cc:240] Negotiation successful
0923 17:28:54.513933 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6766,"thread_start_us":6604,"threads_started":1}
I20260923 17:28:54.514331 15597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:54.483510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:55029 (local address 127.8.29.252:42867)
0923 17:28:54.487040 (+ 3530us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:54.487046 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:54.494054 (+ 7008us) server_negotiation.cc:407] Connection header received
0923 17:28:54.494225 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:54.494231 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:54.494314 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:54.494426 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:54.496387 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.497503 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.499370 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:54.500170 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:54.500292 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:54.506442 (+ 6150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:54.506448 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:54.506452 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:54.506535 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:54.510903 (+ 4368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.511842 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.511848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.511852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.512051 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:54.512736 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:54.512741 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:54.512744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:54.513028 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:54.513170 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:54.513721 (+ 551us) server_negotiation.cc:299] Negotiation successful
0923 17:28:54.513932 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3404,"thread_start_us":3267,"threads_started":1}
I20260923 17:28:54.515492 15587 heartbeater.cc:347] Connected to a master server at 127.8.29.252:42867
I20260923 17:28:54.515652 15587 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:54.516083 15587 heartbeater.cc:511] Master 127.8.29.252:42867 requested a full tablet report, sending...
I20260923 17:28:54.517234 15263 ts_manager.cc:194] Registered new tserver with Master: d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613)
I20260923 17:28:54.600370 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 14804
I20260923 17:28:54.613304 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:44201
--local_ip_for_outbound_sockets=127.8.29.195
--tserver_master_addrs=127.8.29.254:34369
--webserver_port=44637
--webserver_interface=127.8.29.195
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab
--principal=oryx/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.29.254:34369,127.8.29.253:41173
--tserver_master_addrs=127.8.29.254:34369,127.8.29.253:41173,127.8.29.252:42867 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:28:54.877224 15442 heartbeater.cc:503] Master 127.8.29.254:34369 was elected leader, sending a full tablet report...
W20260923 17:28:54.945637 15600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:54.945922 15600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:54.946015 15600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:54.953608 15600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:54.953727 15600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:54.953840 15600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:28:54.964275 15600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab
--principal=oryx/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:44201
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=44637
--tserver_master_addrs=127.8.29.254:34369,127.8.29.253:41173,127.8.29.252:42867
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:54.965622 15600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:54.966989 15600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:54.980422 15605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:54.980500 15606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:54.980443 15608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:54.981747 15600 server_base.cc:1034] running on GCE node
I20260923 17:28:54.982430 15600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:54.983458 15600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:54.984663 15600 hybrid_clock.cc:648] HybridClock initialized: now 1790184534984608 us; error 53 us; skew 500 ppm
I20260923 17:28:54.984990 15600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:54 dist-test-slave-p7jt krb5kdc[14431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184534, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:54.988294 15600 init.cc:377] Logged in from keytab as oryx/127.8.29.195@KRBTEST.COM (short username oryx)
I20260923 17:28:54.989817 15600 webserver.cc:460] Webserver started at http://127.8.29.195:44637/ using document root <none> and password file <none>
I20260923 17:28:54.990576 15600 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:54.990703 15600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:54.996495 15600 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:28:54.999832 15615 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:55.001497 15600 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:28:55.001705 15600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "b6997baeee22454d9624893c37bca7cf"
format_stamp: "Formatted at 2026-09-23 17:28:51 on dist-test-slave-p7jt"
server_key: "8e82b902c09c5d79219edb53559a6d3f"
server_key_iv: "4139fde87b2fba281f171fda2be93592"
server_key_version: "encryptionkey@0"
I20260923 17:28:55.002813 15600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:55.026821 15600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:55.031002 15600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:55.031258 15600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:55.032259 15600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:28:55.033658 15600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:28:55.033753 15600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:55.033859 15600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:28:55.033926 15600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:55.064674 15600 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:44201
I20260923 17:28:55.064723 15727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:44201 every 8 connection(s)
I20260923 17:28:55.066119 15600 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:28:55.073599 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 15600
Sep 23 17:28:55 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184534, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
Sep 23 17:28:55 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184534, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.252@KRBTEST.COM
Sep 23 17:28:55 dist-test-slave-p7jt krb5kdc[14431](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184534, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.253@KRBTEST.COM
I20260923 17:28:55.113696 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.071054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:45447 (local address 127.8.29.253:41173)
0923 17:28:55.083297 (+ 12243us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.083305 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.083324 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:28:55.083391 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.083396 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.083475 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.083586 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.084640 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.086727 (+ 2087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.097589 (+ 10862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.098522 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.098680 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.102645 (+ 3965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.102654 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.102661 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.102723 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.107381 (+ 4658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.109472 (+ 2091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.109479 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.109483 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.109674 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.110176 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.110182 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.110187 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.111427 (+ 1240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.111601 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.113122 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.113340 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12098,"thread_start_us":12665,"threads_started":1}
I20260923 17:28:55.116405 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.070853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.8.29.195:45447)
0923 17:28:55.071465 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:55.071483 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:55.076205 (+ 4722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:55.083623 (+ 7418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:55.083628 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:55.084163 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:55.084451 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.084463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.087064 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.087069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.088437 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.088453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.098736 (+ 10283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.098744 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.098948 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.099585 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:55.099614 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:55.102380 (+ 2766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:55.107581 (+ 5201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:55.107592 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:55.107611 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:55.108295 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:55.109793 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:55.109799 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:55.109803 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:55.110028 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:55.111572 (+ 1544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:55.111581 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:55.112088 (+ 507us) client_negotiation.cc:769] Sending connection context
0923 17:28:55.112413 (+ 325us) client_negotiation.cc:240] Negotiation successful
0923 17:28:55.113093 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":280,"threads_started":1}
I20260923 17:28:55.118053 15730 heartbeater.cc:347] Connected to a master server at 127.8.29.253:41173
I20260923 17:28:55.118360 15730 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:55.119349 15730 heartbeater.cc:511] Master 127.8.29.253:41173 requested a full tablet report, sending...
I20260923 17:28:55.121136 15075 ts_manager.cc:194] Registered new tserver with Master: b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201)
I20260923 17:28:55.122902 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.088565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43484 (local address 127.8.29.253:41173)
0923 17:28:55.099915 (+ 11350us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.099921 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.100889 (+ 968us) server_negotiation.cc:407] Connection header received
0923 17:28:55.100949 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.100953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.101029 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.101132 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.105736 (+ 4604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.107572 (+ 1836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.111127 (+ 3555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.111889 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.111986 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.115822 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.115829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.115835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.115879 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.119219 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.119919 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.119923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.119926 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.120078 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.120527 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.120532 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.120535 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.120899 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.121206 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.122242 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.122531 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11193,"thread_start_us":11363,"threads_started":1}
I20260923 17:28:55.123626 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.069630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:42867 (local address 127.8.29.195:57789)
0923 17:28:55.070317 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:55.070357 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:55.076718 (+ 6361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:55.077605 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:55.077621 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:55.078430 (+ 809us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:55.080726 (+ 2296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.080737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.085582 (+ 4845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.085587 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.088836 (+ 3249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.088846 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.090375 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.090383 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.090705 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.091397 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:55.091434 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:55.094322 (+ 2888us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:55.110324 (+ 16002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:55.110328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:55.110332 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:55.110853 (+ 521us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:55.121233 (+ 10380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:55.121239 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:55.121243 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:55.121454 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:55.122614 (+ 1160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:55.122619 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:55.122810 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 17:28:55.122959 (+ 149us) client_negotiation.cc:240] Negotiation successful
0923 17:28:55.123229 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":262,"threads_started":1}
I20260923 17:28:55.125368 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.069030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:57789 (local address 127.8.29.252:42867)
0923 17:28:55.069514 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.069520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.070494 (+ 974us) server_negotiation.cc:407] Connection header received
0923 17:28:55.076887 (+ 6393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.076894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.077005 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.077103 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.080895 (+ 3792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.085435 (+ 4540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.089003 (+ 3568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.089882 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.090049 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.094975 (+ 4926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.094982 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.094986 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.095027 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.108881 (+ 13854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.111018 (+ 2137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.111023 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.111026 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.111185 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.121641 (+ 10456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.121647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.121651 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.122019 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.122161 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.123965 (+ 1804us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.124166 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":239,"threads_started":1}
I20260923 17:28:55.126082 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.087812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49240 (local address 127.8.29.254:34369)
0923 17:28:55.098300 (+ 10488us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.098307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.098330 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:28:55.098394 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.098401 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.098508 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.099062 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.103771 (+ 4709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.105103 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.109447 (+ 4344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.110228 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.110368 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.118892 (+ 8524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.118900 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.118905 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.118948 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.123194 (+ 4246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.124011 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.124017 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.124020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.124179 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.124547 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.124552 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.124555 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.124913 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.125107 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.125536 (+ 429us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.125767 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9706,"thread_start_us":9570,"threads_started":1}
I20260923 17:28:55.127714 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.071545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.195:37731)
0923 17:28:55.072047 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:55.072064 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:55.076640 (+ 4576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:55.078541 (+ 1901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:55.078545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:55.079131 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:28:55.080315 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.080347 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.081999 (+ 1652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.082004 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.085227 (+ 3223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.085241 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.086652 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.086661 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.086977 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.089539 (+ 2562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:28:55.089589 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:28:55.094745 (+ 5156us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:28:55.124556 (+ 29811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:55.124562 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:55.124566 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:55.124924 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:55.125385 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:28:55.125389 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:28:55.125393 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:28:55.125792 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:28:55.126727 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:28:55.126732 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:28:55.126877 (+ 145us) client_negotiation.cc:769] Sending connection context
0923 17:28:55.126995 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:28:55.127297 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":252,"threads_started":1}
I20260923 17:28:55.127715 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.071692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:37731 (local address 127.8.29.254:34369)
0923 17:28:55.074812 (+ 3120us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.074818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.074840 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:28:55.078293 (+ 3453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.078299 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.078382 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.078518 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.080501 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.081847 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.085721 (+ 3874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.086369 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.086523 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.094959 (+ 8436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.094968 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.094973 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.095021 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.124427 (+ 29406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.125162 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.125165 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.125167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.125273 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.126289 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.126292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.126294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.126621 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.126759 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.127131 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.127335 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3005,"thread_start_us":2911,"threads_started":1}
I20260923 17:28:55.129297 15729 heartbeater.cc:347] Connected to a master server at 127.8.29.254:34369
I20260923 17:28:55.129451 15729 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:55.130080 15729 heartbeater.cc:511] Master 127.8.29.254:34369 requested a full tablet report, sending...
I20260923 17:28:55.130699 15728 heartbeater.cc:347] Connected to a master server at 127.8.29.252:42867
I20260923 17:28:55.130824 15728 heartbeater.cc:464] Registering TS with master...
I20260923 17:28:55.131229 15728 heartbeater.cc:511] Master 127.8.29.252:42867 requested a full tablet report, sending...
I20260923 17:28:55.131232 14476 ts_manager.cc:194] Re-registered known tserver with Master: b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201)
I20260923 17:28:55.132339 14476 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:37731
I20260923 17:28:55.132422 15263 ts_manager.cc:194] Registered new tserver with Master: b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201)
I20260923 17:28:55.131569 14475 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49240:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:55.140861 14475 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:55.142383 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.101160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44798 (local address 127.8.29.252:42867)
0923 17:28:55.125621 (+ 24461us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.125626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.125643 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:28:55.125692 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.125697 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.125800 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.129541 (+ 3741us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.129666 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.130388 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.132434 (+ 2046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.133448 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.133580 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.137030 (+ 3450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.137036 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.137039 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.137074 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.139565 (+ 2491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.140394 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.140399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.140401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.140553 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.141032 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.141038 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.141041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.141361 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.141561 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.141945 (+ 384us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.142119 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24358,"thread_start_us":27989,"threads_started":1}
I20260923 17:28:55.198848 15763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.161785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34613 (local address 127.0.0.1:41564)
0923 17:28:55.190258 (+ 28473us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:55.190276 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:55.190381 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:55.191814 (+ 1433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:55.191819 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:55.191845 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:55.192102 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.192111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.193520 (+ 1409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.193525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.195648 (+ 2123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.195664 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.197524 (+ 1860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.197530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.197710 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.197784 (+ 74us) client_negotiation.cc:769] Sending connection context
0923 17:28:55.197905 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 17:28:55.198162 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28331,"mutex_wait_us":36,"thread_start_us":28224,"threads_started":1}
I20260923 17:28:55.199071 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.162165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.8.29.194:34613)
0923 17:28:55.190988 (+ 28823us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.191000 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.191038 (+ 38us) server_negotiation.cc:407] Connection header received
0923 17:28:55.191161 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.191171 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.191523 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.191761 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:55.192271 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.193356 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.195835 (+ 2479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.197297 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.197437 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.197678 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.198589 (+ 911us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.198783 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28621,"thread_start_us":28500,"threads_started":1}
I20260923 17:28:55.199970 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.161066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:35875 (local address 127.0.0.1:60334)
0923 17:28:55.161785 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:55.161805 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:55.161928 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:55.193107 (+ 31179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:55.193112 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:55.193138 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:55.193438 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.193450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.195291 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.195296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.197356 (+ 2060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.197371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.199244 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.199251 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.199444 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.199488 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 17:28:55.199556 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 17:28:55.199690 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":426,"threads_started":1}
I20260923 17:28:55.201021 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.161335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60334 (local address 127.8.29.193:35875)
0923 17:28:55.192389 (+ 31054us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.192400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.192434 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:28:55.192556 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.192565 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.192868 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.193093 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:55.193620 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.195160 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.197560 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.198955 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.200285 (+ 1330us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.200502 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.200582 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.200746 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30848,"thread_start_us":31905,"threads_started":1}
I20260923 17:28:55.203470 15522 tablet_service.cc:1467] Processing CreateTablet for tablet 65fc22b0408f4d7e82c5189276559c45 (DEFAULT_TABLE table=test-table [id=bc6ac11228094374886df1f3fbcfe9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:55.204983 15522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fc22b0408f4d7e82c5189276559c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:55.204849 15376 tablet_service.cc:1467] Processing CreateTablet for tablet 65fc22b0408f4d7e82c5189276559c45 (DEFAULT_TABLE table=test-table [id=bc6ac11228094374886df1f3fbcfe9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:55.206243 15376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fc22b0408f4d7e82c5189276559c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:55.206511 15762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.161752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:44201 (local address 127.0.0.1:40568)
0923 17:28:55.189555 (+ 27803us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:55.189586 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:55.189746 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:55.195330 (+ 5584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:55.195335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:55.195365 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:55.195910 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.195919 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.197259 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.197265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.200277 (+ 3012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.200290 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.205397 (+ 5107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.205403 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.205626 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.205661 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:28:55.205775 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 17:28:55.205999 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27622,"thread_start_us":28731,"threads_started":1}
I20260923 17:28:55.206533 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.162396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.8.29.195:44201)
0923 17:28:55.193906 (+ 31510us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.193912 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.193937 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:28:55.194007 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.194012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.194167 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.194296 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:55.196094 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.197111 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.200444 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.202034 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.202168 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.202346 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.205881 (+ 3535us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.206083 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31366,"thread_start_us":31199,"threads_started":1}
I20260923 17:28:55.210918 15663 tablet_service.cc:1467] Processing CreateTablet for tablet 65fc22b0408f4d7e82c5189276559c45 (DEFAULT_TABLE table=test-table [id=bc6ac11228094374886df1f3fbcfe9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:55.212376 15663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 65fc22b0408f4d7e82c5189276559c45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:55.220254 15766 tablet_bootstrap.cc:492] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: Bootstrap starting.
I20260923 17:28:55.223610 15767 tablet_bootstrap.cc:492] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: Bootstrap starting.
I20260923 17:28:55.223677 15766 tablet_bootstrap.cc:654] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:55.225211 15766 log.cc:824] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:55.227209 15767 tablet_bootstrap.cc:654] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:55.228488 15769 tablet_bootstrap.cc:492] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: Bootstrap starting.
I20260923 17:28:55.228533 15767 log.cc:824] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:55.228612 15766 tablet_bootstrap.cc:492] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: No bootstrap required, opened a new log
I20260923 17:28:55.228873 15766 ts_tablet_manager.cc:1397] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 17:28:55.232926 15767 tablet_bootstrap.cc:492] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: No bootstrap required, opened a new log
I20260923 17:28:55.233152 15769 tablet_bootstrap.cc:654] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:55.233232 15767 ts_tablet_manager.cc:1397] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260923 17:28:55.234177 15769 log.cc:824] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:55.236788 15769 tablet_bootstrap.cc:492] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: No bootstrap required, opened a new log
I20260923 17:28:55.237038 15769 ts_tablet_manager.cc:1397] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 17:28:55.239845 15766 raft_consensus.cc:348] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } }
I20260923 17:28:55.240316 15766 raft_consensus.cc:374] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:55.240412 15766 raft_consensus.cc:729] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c0e000eae64596a8d6eb329c4598b9, State: Initialized, Role: FOLLOWER
I20260923 17:28:55.242784 15766 consensus_queue.cc:260] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } }
I20260923 17:28:55.243582 15767 raft_consensus.cc:348] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } }
I20260923 17:28:55.244020 15767 raft_consensus.cc:374] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:55.244112 15767 raft_consensus.cc:729] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455cb603b43346f0b1d56e6dc06f589b, State: Initialized, Role: FOLLOWER
I20260923 17:28:55.244800 15767 consensus_queue.cc:260] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } }
I20260923 17:28:55.245342 15769 raft_consensus.cc:348] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } }
I20260923 17:28:55.245927 15769 raft_consensus.cc:374] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:55.246021 15769 raft_consensus.cc:729] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6997baeee22454d9624893c37bca7cf, State: Initialized, Role: FOLLOWER
I20260923 17:28:55.246840 15769 consensus_queue.cc:260] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } }
I20260923 17:28:55.264540 15766 ts_tablet_manager.cc:1428] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: Time spent starting tablet: real 0.036s user 0.021s sys 0.012s
I20260923 17:28:55.265024 15588 heartbeater.cc:503] Master 127.8.29.254:34369 was elected leader, sending a full tablet report...
I20260923 17:28:55.265174 15767 ts_tablet_manager.cc:1428] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: Time spent starting tablet: real 0.032s user 0.030s sys 0.003s
I20260923 17:28:55.266227 15773 raft_consensus.cc:482] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:55.266623 15769 ts_tablet_manager.cc:1428] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: Time spent starting tablet: real 0.029s user 0.013s sys 0.015s
I20260923 17:28:55.268426 15729 heartbeater.cc:503] Master 127.8.29.254:34369 was elected leader, sending a full tablet report...
I20260923 17:28:55.269506 15773 raft_consensus.cc:504] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } }
I20260923 17:28:55.271494 15773 leader_election.cc:288] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201), d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613)
I20260923 17:28:55.283773 15764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.272366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:36967 (local address 127.8.29.194:34613)
0923 17:28:55.272540 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.272546 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.276956 (+ 4410us) server_negotiation.cc:407] Connection header received
0923 17:28:55.277137 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.277142 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.277215 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.277307 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:55.278082 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.279141 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.281481 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.282857 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.283010 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.283066 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.283475 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.283579 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:28:55.283942 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.271865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34613 (local address 127.8.29.193:36967)
0923 17:28:55.276853 (+ 4988us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:55.276875 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:55.277024 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:55.277442 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:55.277447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:55.277477 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:55.277900 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.277918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.279286 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.279292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.281310 (+ 2018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.281323 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.283093 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.283100 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.283291 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.283328 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:28:55.283435 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 17:28:55.283608 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4821,"thread_start_us":4644,"threads_started":1}
I20260923 17:28:55.285106 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.272550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:53791 (local address 127.8.29.195:44201)
0923 17:28:55.272756 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.272761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.278027 (+ 5266us) server_negotiation.cc:407] Connection header received
0923 17:28:55.279096 (+ 1069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.279102 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.279177 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.279232 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:55.279949 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.280757 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.283040 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.284213 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.284368 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.284437 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.284772 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.284871 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:28:55.285307 15777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.272111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:44201 (local address 127.8.29.193:53791)
0923 17:28:55.277940 (+ 5829us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:55.277962 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:55.278117 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:55.279403 (+ 1286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:55.279406 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:55.279431 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:55.279766 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.279778 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.280899 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.280903 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.282841 (+ 1938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:55.282854 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.284455 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.284461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:55.284596 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.284627 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:28:55.284722 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:28:55.284851 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5690,"thread_start_us":5554,"threads_started":1}
I20260923 17:28:55.285144 15542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fc22b0408f4d7e82c5189276559c45" candidate_uuid: "455cb603b43346f0b1d56e6dc06f589b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c0e000eae64596a8d6eb329c4598b9" is_pre_election: true
I20260923 17:28:55.285728 15542 raft_consensus.cc:2454] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 455cb603b43346f0b1d56e6dc06f589b in term 0.
I20260923 17:28:55.286410 15331 leader_election.cc:302] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 455cb603b43346f0b1d56e6dc06f589b, d7c0e000eae64596a8d6eb329c4598b9; no voters:
I20260923 17:28:55.286355 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fc22b0408f4d7e82c5189276559c45" candidate_uuid: "455cb603b43346f0b1d56e6dc06f589b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6997baeee22454d9624893c37bca7cf" is_pre_election: true
I20260923 17:28:55.286849 15775 raft_consensus.cc:2781] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:55.286835 15683 raft_consensus.cc:2454] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 455cb603b43346f0b1d56e6dc06f589b in term 0.
I20260923 17:28:55.287078 15775 raft_consensus.cc:482] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:55.287240 15775 raft_consensus.cc:3037] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:55.289521 15775 raft_consensus.cc:504] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } }
I20260923 17:28:55.290169 15775 leader_election.cc:288] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [CANDIDATE]: Term 1 election: Requested vote from peers b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201), d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613)
I20260923 17:28:55.290580 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fc22b0408f4d7e82c5189276559c45" candidate_uuid: "455cb603b43346f0b1d56e6dc06f589b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6997baeee22454d9624893c37bca7cf"
I20260923 17:28:55.290745 15542 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "65fc22b0408f4d7e82c5189276559c45" candidate_uuid: "455cb603b43346f0b1d56e6dc06f589b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c0e000eae64596a8d6eb329c4598b9"
I20260923 17:28:55.290784 15683 raft_consensus.cc:3037] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:55.290931 15542 raft_consensus.cc:3037] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:55.293249 15683 raft_consensus.cc:2454] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 455cb603b43346f0b1d56e6dc06f589b in term 1.
I20260923 17:28:55.293576 15542 raft_consensus.cc:2454] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 455cb603b43346f0b1d56e6dc06f589b in term 1.
I20260923 17:28:55.293629 15329 leader_election.cc:302] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 455cb603b43346f0b1d56e6dc06f589b, b6997baeee22454d9624893c37bca7cf; no voters:
I20260923 17:28:55.293883 15775 raft_consensus.cc:2781] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:55.294056 15775 raft_consensus.cc:686] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 1 LEADER]: Becoming Leader. State: Replica: 455cb603b43346f0b1d56e6dc06f589b, State: Running, Role: LEADER
I20260923 17:28:55.294596 15775 consensus_queue.cc:237] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } }
I20260923 17:28:55.298982 14475 catalog_manager.cc:5495] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b reported cstate change: term changed from 0 to 1, leader changed from <none> to 455cb603b43346f0b1d56e6dc06f589b (127.8.29.193). New cstate: current_term: 1 leader_uuid: "455cb603b43346f0b1d56e6dc06f589b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } health_report { overall_health: UNKNOWN } } }
W20260923 17:28:55.319327 15731 tablet.cc:2369] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:28:55.334252 15444 tablet.cc:2369] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:55.335863 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.330507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60346 (local address 127.8.29.193:35875)
0923 17:28:55.330709 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.330716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.330779 (+ 63us) server_negotiation.cc:407] Connection header received
0923 17:28:55.330976 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.330979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.331069 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.331195 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:55.331500 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.332192 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.333675 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.334119 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.334219 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.334799 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:55.335318 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:55.335430 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.335528 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.335612 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:28:55.344309 15683 raft_consensus.cc:1261] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [term 1 FOLLOWER]: Refusing update from remote peer 455cb603b43346f0b1d56e6dc06f589b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:55.344309 15542 raft_consensus.cc:1261] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [term 1 FOLLOWER]: Refusing update from remote peer 455cb603b43346f0b1d56e6dc06f589b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:55.345276 15773 consensus_queue.cc:1035] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:55.345616 15775 consensus_queue.cc:1035] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:55.361021 15781 mvcc.cc:205] Tried to move back new op lower bound from 7332595856759775232 to 7332595856570241024. Current Snapshot: MvccSnapshot[applied={T|T < 7332595856759775232}]
I20260923 17:28:55.378860 14475 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:55.379154 14475 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49240:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:55.386755 14475 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6893904a76694629b825a55109145c87: Sending DeleteTablet for 3 replicas of tablet 65fc22b0408f4d7e82c5189276559c45
I20260923 17:28:55.387921 15663 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fc22b0408f4d7e82c5189276559c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:55 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:40568
I20260923 17:28:55.387885 15376 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fc22b0408f4d7e82c5189276559c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:55 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:60334
I20260923 17:28:55.388281 15522 tablet_service.cc:1514] Processing DeleteTablet for tablet 65fc22b0408f4d7e82c5189276559c45 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:55 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:41564
I20260923 17:28:55.403679 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 14444
I20260923 17:28:55.404021 15793 tablet_replica.cc:330] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: stopping tablet replica
I20260923 17:28:55.404498 15793 raft_consensus.cc:2229] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:55.405016 14506 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:55.405023 15793 raft_consensus.cc:2258] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:55.407524 15793 ts_tablet_manager.cc:1905] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:55.408205 15792 tablet_replica.cc:330] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: stopping tablet replica
I20260923 17:28:55.408619 15792 raft_consensus.cc:2229] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:55.409088 15792 raft_consensus.cc:2258] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:55.415302 15791 tablet_replica.cc:330] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: stopping tablet replica
I20260923 17:28:55.415759 15791 raft_consensus.cc:2229] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:55.416527 15791 raft_consensus.cc:2258] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:55.416764 15792 ts_tablet_manager.cc:1905] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:55.420243 15791 ts_tablet_manager.cc:1905] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:55.429144 15791 ts_tablet_manager.cc:1918] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:55.432781 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.390855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43496 (local address 127.8.29.253:41173)
0923 17:28:55.391080 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.391087 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.402118 (+ 11031us) server_negotiation.cc:407] Connection header received
0923 17:28:55.414196 (+ 12078us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.414203 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.414298 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.414419 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.419036 (+ 4617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.420134 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.421972 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.422690 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.422793 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.426332 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.426339 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.426344 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.426388 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.429618 (+ 3230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.430486 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.430491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.430494 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.430655 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.431091 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.431096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.431100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.431424 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.431628 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.432206 (+ 578us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.432439 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:28:55.434619 15793 ts_tablet_manager.cc:1918] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:55.434837 15793 log.cc:1196] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/65fc22b0408f4d7e82c5189276559c45
I20260923 17:28:55.435477 15793 ts_tablet_manager.cc:1939] T 65fc22b0408f4d7e82c5189276559c45 P d7c0e000eae64596a8d6eb329c4598b9: Deleting consensus metadata
I20260923 17:28:55.438622 15791 log.cc:1196] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/65fc22b0408f4d7e82c5189276559c45
I20260923 17:28:55.439245 15791 ts_tablet_manager.cc:1939] T 65fc22b0408f4d7e82c5189276559c45 P 455cb603b43346f0b1d56e6dc06f589b: Deleting consensus metadata
I20260923 17:28:55.445093 15792 ts_tablet_manager.cc:1918] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:55.445282 15792 log.cc:1196] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/65fc22b0408f4d7e82c5189276559c45
I20260923 17:28:55.445850 15792 ts_tablet_manager.cc:1939] T 65fc22b0408f4d7e82c5189276559c45 P b6997baeee22454d9624893c37bca7cf: Deleting consensus metadata
I20260923 17:28:55.450825 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.402848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.8.29.252:42867)
0923 17:28:55.403046 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.403053 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.434707 (+ 31654us) server_negotiation.cc:407] Connection header received
0923 17:28:55.435158 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.435164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.435259 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.435334 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.437223 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.438386 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.440480 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.441159 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.441247 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.444742 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.444748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.444753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.444793 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.448187 (+ 3394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.448887 (+ 700us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.448893 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.448897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.449072 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.449422 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.449427 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.449430 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.449697 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.449950 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.450250 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.450449 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:28:55.559476 14460 catalog_manager.cc:4841] TS b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201): tablet 65fc22b0408f4d7e82c5189276559c45 (table test-table [id=bc6ac11228094374886df1f3fbcfe9a6]) successfully deleted
I20260923 17:28:55.560552 14461 catalog_manager.cc:4841] TS 455cb603b43346f0b1d56e6dc06f589b (127.8.29.193:35875): tablet 65fc22b0408f4d7e82c5189276559c45 (table test-table [id=bc6ac11228094374886df1f3fbcfe9a6]) successfully deleted
I20260923 17:28:55.560598 14462 catalog_manager.cc:4841] TS d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613): tablet 65fc22b0408f4d7e82c5189276559c45 (table test-table [id=bc6ac11228094374886df1f3fbcfe9a6]) successfully deleted
I20260923 17:28:55.561120 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 14444
I20260923 17:28:55.607889 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.593192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44808 (local address 127.8.29.252:42867)
0923 17:28:55.593379 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.593383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.593615 (+ 232us) server_negotiation.cc:407] Connection header received
0923 17:28:55.593766 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.593770 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.593830 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.593907 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.595714 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.596852 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.598591 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.599148 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.599244 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.602609 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.602613 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.602616 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.602648 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.605126 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.605753 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.605759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.605761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.605901 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.606303 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.606309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.606312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.606717 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.606955 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.607325 (+ 370us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.607552 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:28:55.609236 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:55.592152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43502 (local address 127.8.29.253:41173)
0923 17:28:55.592353 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:55.592360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:55.592403 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:28:55.592610 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:55.592617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:55.592729 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:55.592806 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:55.595307 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.596190 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.597815 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:55.598475 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:55.598591 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:55.602333 (+ 3742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:55.602340 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:55.602345 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:55.602385 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:55.605972 (+ 3587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.606714 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.606720 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.606724 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.606875 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:55.607402 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:55.607407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:55.607410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:55.607776 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:55.607974 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:55.608524 (+ 550us) server_negotiation.cc:299] Negotiation successful
0923 17:28:55.608750 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:28:56.365110 15802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.364301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.195:33821)
0923 17:28:56.364778 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.364902 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":258,"threads_started":1}
W20260923 17:28:56.365491 15729 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:34369 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
I20260923 17:28:56.368085 15803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.367263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.194:58813)
0923 17:28:56.367772 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.367896 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":342,"threads_started":1}
W20260923 17:28:56.368866 15588 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:34369 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
I20260923 17:28:56.373603 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.372855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.193:55401)
0923 17:28:56.373285 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.373434 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":206,"threads_started":1}
W20260923 17:28:56.373927 15442 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:34369 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
I20260923 17:28:56.886386 15805 raft_consensus.cc:482] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6893904a76694629b825a55109145c87)
I20260923 17:28:56.886761 15805 raft_consensus.cc:504] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } }
I20260923 17:28:56.886895 15806 raft_consensus.cc:482] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 6893904a76694629b825a55109145c87)
I20260923 17:28:56.887068 15806 raft_consensus.cc:504] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } }
I20260923 17:28:56.887856 15806 leader_election.cc:288] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6893904a76694629b825a55109145c87 (127.8.29.254:34369), c001745d65fc469b9b595186ec0be467 (127.8.29.252:42867)
I20260923 17:28:56.888258 15805 leader_election.cc:288] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 6893904a76694629b825a55109145c87 (127.8.29.254:34369), b894fbfff63a4a1297abadee53713db3 (127.8.29.253:41173)
I20260923 17:28:56.889227 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.888103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52174)
0923 17:28:56.888702 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.888923 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":315,"threads_started":1}
I20260923 17:28:56.895704 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.888359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52190)
0923 17:28:56.895215 (+ 6856us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.895434 (+ 219us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":6710,"thread_start_us":6554,"threads_started":1}
W20260923 17:28:56.895761 15059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
W20260923 17:28:56.896835 15246 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
I20260923 17:28:56.898588 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.897418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52204)
0923 17:28:56.898054 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.898293 (+ 239us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393}
W20260923 17:28:56.898753 15059 leader_election.cc:334] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6893904a76694629b825a55109145c87 (127.8.29.254:34369): Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
I20260923 17:28:56.899977 15814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.898814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52218)
0923 17:28:56.899539 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.899722 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":430,"threads_started":1}
W20260923 17:28:56.900203 15246 leader_election.cc:334] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 6893904a76694629b825a55109145c87 (127.8.29.254:34369): Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
I20260923 17:28:56.902365 15810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.888992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.8.29.252:42867)
0923 17:28:56.895004 (+ 6012us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:56.895010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:56.895037 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:28:56.895119 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:56.895125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:56.895222 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:56.896300 (+ 1078us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:56.896596 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.897867 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.900324 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.901540 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.901657 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:56.901748 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:56.902017 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 17:28:56.902124 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":5856,"spinlock_wait_cycles":13568,"thread_start_us":5686,"threads_started":1}
I20260923 17:28:56.902490 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.888771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:42867 (local address 127.0.0.1:51080)
0923 17:28:56.889535 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.889553 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:56.889698 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:56.895691 (+ 5993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:56.895697 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:56.895732 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:56.896013 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:56.896028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.898056 (+ 2028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.898061 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:56.900145 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:56.900161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.901688 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.901694 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:56.901859 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:56.901886 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:28:56.901987 (+ 101us) client_negotiation.cc:240] Negotiation successful
0923 17:28:56.902158 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":6655,"threads_started":1}
I20260923 17:28:56.903086 15273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b894fbfff63a4a1297abadee53713db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c001745d65fc469b9b595186ec0be467" is_pre_election: true
I20260923 17:28:56.903463 15273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b894fbfff63a4a1297abadee53713db3 in term 1.
I20260923 17:28:56.903965 15059 leader_election.cc:302] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b894fbfff63a4a1297abadee53713db3, c001745d65fc469b9b595186ec0be467; no voters: 6893904a76694629b825a55109145c87
I20260923 17:28:56.904191 15808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.896191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:41173 (local address 127.0.0.1:45044)
0923 17:28:56.896314 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.896330 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:56.896447 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:56.896928 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:56.896933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:56.896970 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:56.897356 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:56.897369 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.899251 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.899257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:56.901828 (+ 2571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:56.901839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.903394 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.903401 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:56.903665 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:56.903698 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:28:56.903797 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:28:56.903932 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":587}
I20260923 17:28:56.904242 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.895450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.8.29.253:41173)
0923 17:28:56.896264 (+ 814us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:56.896271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:56.896610 (+ 339us) server_negotiation.cc:407] Connection header received
0923 17:28:56.896680 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:56.896685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:56.896755 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:56.897499 (+ 744us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:56.897729 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.899095 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.901973 (+ 2878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.903153 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.903244 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:56.903399 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:56.903825 (+ 426us) server_negotiation.cc:299] Negotiation successful
0923 17:28:56.903944 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":685,"thread_start_us":444,"threads_started":1}
I20260923 17:28:56.904482 15806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 17:28:56.904613 15806 raft_consensus.cc:482] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 6893904a76694629b825a55109145c87)
I20260923 17:28:56.904718 15806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:28:56.904912 15085 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c001745d65fc469b9b595186ec0be467" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b894fbfff63a4a1297abadee53713db3" is_pre_election: true
I20260923 17:28:56.906980 15806 raft_consensus.cc:504] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } }
I20260923 17:28:56.907401 15085 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c001745d65fc469b9b595186ec0be467 in current term 2: Already voted for candidate b894fbfff63a4a1297abadee53713db3 in this term.
I20260923 17:28:56.907617 15806 leader_election.cc:288] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [CANDIDATE]: Term 2 election: Requested vote from peers 6893904a76694629b825a55109145c87 (127.8.29.254:34369), c001745d65fc469b9b595186ec0be467 (127.8.29.252:42867)
I20260923 17:28:56.908303 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.907855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52238)
0923 17:28:56.908055 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.908147 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:28:56.908521 15246 leader_election.cc:302] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c001745d65fc469b9b595186ec0be467; no voters: 6893904a76694629b825a55109145c87, b894fbfff63a4a1297abadee53713db3
I20260923 17:28:56.908563 15273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b894fbfff63a4a1297abadee53713db3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "c001745d65fc469b9b595186ec0be467"
I20260923 17:28:56.908758 15273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:28:56.911077 15273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b894fbfff63a4a1297abadee53713db3 in term 2.
I20260923 17:28:56.911263 15805 raft_consensus.cc:2726] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:28:56.911581 15059 leader_election.cc:302] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b894fbfff63a4a1297abadee53713db3, c001745d65fc469b9b595186ec0be467; no voters:
I20260923 17:28:56.911856 15806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:28:56.912353 15806 raft_consensus.cc:686] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [term 2 LEADER]: Becoming Leader. State: Replica: b894fbfff63a4a1297abadee53713db3, State: Running, Role: LEADER
I20260923 17:28:56.912425 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.911951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52240)
0923 17:28:56.912103 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.912231 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 17:28:56.912569 15059 leader_election.cc:334] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 6893904a76694629b825a55109145c87 (127.8.29.254:34369): Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
I20260923 17:28:56.912807 15806 consensus_queue.cc:237] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } }
I20260923 17:28:56.914546 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b894fbfff63a4a1297abadee53713db3. Latest consensus state: current_term: 2 leader_uuid: "b894fbfff63a4a1297abadee53713db3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:56.914840 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:56.915386 15817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:56.918905 15817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:56.919710 15817 catalog_manager.cc:1259] Loaded cluster ID: d7687137e2964f55b9c555ae17474c0a
I20260923 17:28:56.919796 15817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:56.920888 15817 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:56.921653 15817 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Loaded TSK: 0
I20260923 17:28:56.922487 15817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:28:56.956697 15075 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:28:56.963860 15075 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:28:56.968597 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.968132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52260)
0923 17:28:56.968312 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.968413 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 17:28:56.968561 15273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [term 2 FOLLOWER]: Refusing update from remote peer b894fbfff63a4a1297abadee53713db3: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 17:28:56.969326 15815 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260923 17:28:56.969735 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.969344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52262)
0923 17:28:56.969487 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.969554 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:28:56.969888 15059 consensus_peers.cc:487] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 -> Peer 6893904a76694629b825a55109145c87 (127.8.29.254:34369): Couldn't send request to peer 6893904a76694629b825a55109145c87. Status: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 17:28:56.972374 15805 sys_catalog.cc:455] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b894fbfff63a4a1297abadee53713db3. Latest consensus state: current_term: 2 leader_uuid: "b894fbfff63a4a1297abadee53713db3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:56.972690 15805 sys_catalog.cc:458] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:28:56.974148 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "b894fbfff63a4a1297abadee53713db3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:56.974476 15805 sys_catalog.cc:455] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "b894fbfff63a4a1297abadee53713db3" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "6893904a76694629b825a55109145c87" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 34369 } } peers { permanent_uuid: "b894fbfff63a4a1297abadee53713db3" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 41173 } attrs { promote: false } } peers { permanent_uuid: "c001745d65fc469b9b595186ec0be467" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 42867 } attrs { promote: false } } }
I20260923 17:28:56.974725 15805 sys_catalog.cc:458] T 00000000000000000000000000000000 P c001745d65fc469b9b595186ec0be467 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:28:56.974424 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:56.991015 15825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.983896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59990 (local address 127.8.29.195:44201)
0923 17:28:56.984329 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:56.984335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:56.985337 (+ 1002us) server_negotiation.cc:407] Connection header received
0923 17:28:56.985471 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:56.985477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:56.985569 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:56.985655 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:56.986133 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.986993 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.989178 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.989942 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.990073 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:56.990191 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:56.990630 (+ 439us) server_negotiation.cc:299] Negotiation successful
0923 17:28:56.990775 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":539,"threads_started":1}
I20260923 17:28:56.992041 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.984663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:44201 (local address 127.0.0.1:59990)
0923 17:28:56.985237 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.985250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:56.985356 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:56.985671 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:56.985676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:56.985698 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:56.985988 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:56.985997 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.987142 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.987147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:56.989031 (+ 1884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:56.989045 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.990208 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.990215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:56.990394 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:56.990428 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:28:56.991572 (+ 1144us) client_negotiation.cc:240] Negotiation successful
0923 17:28:56.991711 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":482,"thread_start_us":244,"threads_started":1}
I20260923 17:28:56.992985 15663 tablet_service.cc:1467] Processing CreateTablet for tablet c87f2ba6f8c54113b7af9b505f16d8b3 (DEFAULT_TABLE table=test-table [id=5faed8ffc54a4ab29a327b88752f6588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:56.993538 15663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87f2ba6f8c54113b7af9b505f16d8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:57.000524 15809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.983589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:35875 (local address 127.0.0.1:59590)
0923 17:28:56.983792 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.983809 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:56.983909 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:56.995125 (+ 11216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:56.995130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:56.995157 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:56.995415 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:56.995424 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.996746 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.996750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:56.998006 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:56.998019 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.999647 (+ 1628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.999654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:56.999857 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:56.999901 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 17:28:56.999975 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:28:57.000145 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 17:28:57.001721 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.983925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59590 (local address 127.8.29.193:35875)
0923 17:28:56.994802 (+ 10877us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:56.994807 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:56.994824 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:28:56.994903 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:56.994906 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:56.994973 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:56.995108 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:56.995564 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.996589 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:56.998174 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:56.999367 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.001035 (+ 1668us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:57.001151 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:57.001211 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 17:28:57.001374 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10773,"thread_start_us":10681,"threads_started":1}
I20260923 17:28:57.002604 15376 tablet_service.cc:1467] Processing CreateTablet for tablet c87f2ba6f8c54113b7af9b505f16d8b3 (DEFAULT_TABLE table=test-table [id=5faed8ffc54a4ab29a327b88752f6588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:57.003123 15376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87f2ba6f8c54113b7af9b505f16d8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:57.006422 15828 tablet_bootstrap.cc:492] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: Bootstrap starting.
I20260923 17:28:57.008347 15828 tablet_bootstrap.cc:654] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:57.010310 15830 tablet_bootstrap.cc:492] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: Bootstrap starting.
I20260923 17:28:57.010586 15828 tablet_bootstrap.cc:492] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: No bootstrap required, opened a new log
I20260923 17:28:57.010736 15828 ts_tablet_manager.cc:1397] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20260923 17:28:57.011459 15807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.982708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34613 (local address 127.0.0.1:45186)
0923 17:28:56.982884 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:56.982902 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:28:56.983024 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:28:57.005521 (+ 22497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:28:57.005528 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:28:57.005577 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:28:57.005786 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:57.005798 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.007222 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.007228 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:57.009009 (+ 1781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:28:57.009025 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.010716 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.010723 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:28:57.010901 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:57.010936 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:28:57.011022 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 17:28:57.011172 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 17:28:57.011484 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:56.982806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.8.29.194:34613)
0923 17:28:57.005145 (+ 22339us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:57.005152 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:57.005174 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:28:57.005249 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:57.005254 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:57.005332 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:57.005451 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:28:57.005953 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.007069 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.009250 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.010491 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.010758 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:57.010910 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:57.011063 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 17:28:57.011203 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22198,"thread_start_us":22746,"threads_started":1}
I20260923 17:28:57.011583 15828 raft_consensus.cc:348] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } }
I20260923 17:28:57.011830 15828 raft_consensus.cc:374] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:57.011925 15828 raft_consensus.cc:729] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6997baeee22454d9624893c37bca7cf, State: Initialized, Role: FOLLOWER
I20260923 17:28:57.012228 15830 tablet_bootstrap.cc:654] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:57.012182 15828 consensus_queue.cc:260] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } }
I20260923 17:28:57.012553 15522 tablet_service.cc:1467] Processing CreateTablet for tablet c87f2ba6f8c54113b7af9b505f16d8b3 (DEFAULT_TABLE table=test-table [id=5faed8ffc54a4ab29a327b88752f6588]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:28:57.012995 15828 ts_tablet_manager.cc:1428] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:57.013012 15522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c87f2ba6f8c54113b7af9b505f16d8b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:57.015230 15830 tablet_bootstrap.cc:492] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: No bootstrap required, opened a new log
I20260923 17:28:57.015317 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.013581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.195:47369)
0923 17:28:57.014931 (+ 1350us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.015068 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1207,"thread_start_us":1082,"threads_started":1}
I20260923 17:28:57.015369 15830 ts_tablet_manager.cc:1397] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260923 17:28:57.016161 15830 raft_consensus.cc:348] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } }
I20260923 17:28:57.016382 15830 raft_consensus.cc:374] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:57.016465 15830 raft_consensus.cc:729] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 455cb603b43346f0b1d56e6dc06f589b, State: Initialized, Role: FOLLOWER
I20260923 17:28:57.016690 15830 consensus_queue.cc:260] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } }
I20260923 17:28:57.018028 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.017398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.195:49603)
0923 17:28:57.017800 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.017886 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":293}
I20260923 17:28:57.018626 15830 ts_tablet_manager.cc:1428] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260923 17:28:57.020316 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.019295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.193:53631)
0923 17:28:57.019894 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.020019 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":355,"threads_started":1}
I20260923 17:28:57.021600 15835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.020965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.193:43029)
0923 17:28:57.021332 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.021430 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"threads_started":1}
I20260923 17:28:57.022351 15443 heartbeater.cc:503] Master 127.8.29.253:41173 was elected leader, sending a full tablet report...
I20260923 17:28:57.022635 15836 tablet_bootstrap.cc:492] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: Bootstrap starting.
I20260923 17:28:57.024435 15836 tablet_bootstrap.cc:654] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:57.026448 15836 tablet_bootstrap.cc:492] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: No bootstrap required, opened a new log
I20260923 17:28:57.026633 15836 ts_tablet_manager.cc:1397] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 17:28:57.027256 15836 raft_consensus.cc:348] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } }
I20260923 17:28:57.027462 15836 raft_consensus.cc:374] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:57.027550 15836 raft_consensus.cc:729] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7c0e000eae64596a8d6eb329c4598b9, State: Initialized, Role: FOLLOWER
I20260923 17:28:57.027769 15836 consensus_queue.cc:260] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } }
I20260923 17:28:57.028525 15836 ts_tablet_manager.cc:1428] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:28:57.029748 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.028868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.194:51823)
0923 17:28:57.029328 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.029466 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":220,"threads_started":1}
I20260923 17:28:57.031270 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.030910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.194:44335)
0923 17:28:57.031043 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.031128 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:28:57.221388 15590 tablet.cc:2369] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:28:57.224454 15834 raft_consensus.cc:482] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:28:57.224653 15834 raft_consensus.cc:504] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } }
I20260923 17:28:57.225328 15834 leader_election.cc:288] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613), b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201)
I20260923 17:28:57.225868 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87f2ba6f8c54113b7af9b505f16d8b3" candidate_uuid: "455cb603b43346f0b1d56e6dc06f589b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6997baeee22454d9624893c37bca7cf" is_pre_election: true
I20260923 17:28:57.225868 15541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87f2ba6f8c54113b7af9b505f16d8b3" candidate_uuid: "455cb603b43346f0b1d56e6dc06f589b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c0e000eae64596a8d6eb329c4598b9" is_pre_election: true
I20260923 17:28:57.226128 15683 raft_consensus.cc:2454] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 455cb603b43346f0b1d56e6dc06f589b in term 0.
I20260923 17:28:57.226169 15541 raft_consensus.cc:2454] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 455cb603b43346f0b1d56e6dc06f589b in term 0.
I20260923 17:28:57.226603 15329 leader_election.cc:302] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 455cb603b43346f0b1d56e6dc06f589b, b6997baeee22454d9624893c37bca7cf; no voters:
I20260923 17:28:57.226863 15834 raft_consensus.cc:2781] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:28:57.226976 15834 raft_consensus.cc:482] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:28:57.227049 15834 raft_consensus.cc:3037] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:57.229005 15834 raft_consensus.cc:504] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } }
I20260923 17:28:57.229523 15834 leader_election.cc:288] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [CANDIDATE]: Term 1 election: Requested vote from peers d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613), b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201)
I20260923 17:28:57.230027 15541 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87f2ba6f8c54113b7af9b505f16d8b3" candidate_uuid: "455cb603b43346f0b1d56e6dc06f589b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7c0e000eae64596a8d6eb329c4598b9"
I20260923 17:28:57.230060 15683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c87f2ba6f8c54113b7af9b505f16d8b3" candidate_uuid: "455cb603b43346f0b1d56e6dc06f589b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b6997baeee22454d9624893c37bca7cf"
I20260923 17:28:57.230257 15541 raft_consensus.cc:3037] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:57.230263 15683 raft_consensus.cc:3037] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:57.233129 15683 raft_consensus.cc:2454] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 455cb603b43346f0b1d56e6dc06f589b in term 1.
I20260923 17:28:57.233130 15541 raft_consensus.cc:2454] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 455cb603b43346f0b1d56e6dc06f589b in term 1.
I20260923 17:28:57.233614 15329 leader_election.cc:302] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 455cb603b43346f0b1d56e6dc06f589b, b6997baeee22454d9624893c37bca7cf, d7c0e000eae64596a8d6eb329c4598b9; no voters:
I20260923 17:28:57.233913 15834 raft_consensus.cc:2781] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:57.234310 15834 raft_consensus.cc:686] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 1 LEADER]: Becoming Leader. State: Replica: 455cb603b43346f0b1d56e6dc06f589b, State: Running, Role: LEADER
I20260923 17:28:57.234822 15834 consensus_queue.cc:237] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } }
I20260923 17:28:57.235443 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.234925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.193:46737)
0923 17:28:57.235120 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.235263 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:28:57.237498 15074 catalog_manager.cc:5495] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b reported cstate change: term changed from 0 to 1, leader changed from <none> to 455cb603b43346f0b1d56e6dc06f589b (127.8.29.193). New cstate: current_term: 1 leader_uuid: "455cb603b43346f0b1d56e6dc06f589b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "455cb603b43346f0b1d56e6dc06f589b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35875 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 } health_report { overall_health: UNKNOWN } } }
I20260923 17:28:57.293111 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.288321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59598 (local address 127.8.29.193:35875)
0923 17:28:57.288502 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:57.288506 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:57.288524 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:28:57.288606 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:57.288609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:57.288672 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:57.288740 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:28:57.289119 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.289887 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.291371 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.291811 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.291944 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:57.292367 (+ 423us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:28:57.292487 (+ 120us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:28:57.292573 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:57.292729 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 17:28:57.292814 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:28:57.296283 15683 raft_consensus.cc:1261] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [term 1 FOLLOWER]: Refusing update from remote peer 455cb603b43346f0b1d56e6dc06f589b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:57.296283 15541 raft_consensus.cc:1261] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [term 1 FOLLOWER]: Refusing update from remote peer 455cb603b43346f0b1d56e6dc06f589b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:28:57.296902 15841 consensus_queue.cc:1035] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7c0e000eae64596a8d6eb329c4598b9" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:57.297228 15834 consensus_queue.cc:1035] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b6997baeee22454d9624893c37bca7cf" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:28:57.298985 15589 heartbeater.cc:503] Master 127.8.29.253:41173 was elected leader, sending a full tablet report...
I20260923 17:28:57.298995 15730 heartbeater.cc:503] Master 127.8.29.253:41173 was elected leader, sending a full tablet report...
I20260923 17:28:57.300020 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.299490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.195:35579)
0923 17:28:57.299719 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.299834 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260923 17:28:57.300972 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.300451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.194:47307)
0923 17:28:57.300629 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.300758 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 17:28:57.307391 15840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.306308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.194:42217)
0923 17:28:57.307105 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.307218 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":687}
I20260923 17:28:57.308512 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.308028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.195:52483)
0923 17:28:57.308245 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.308334 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260923 17:28:57.311429 15846 mvcc.cc:205] Tried to move back new op lower bound from 7332595864759562240 to 7332595864516042752. Current Snapshot: MvccSnapshot[applied={T|T < 7332595864759562240}]
I20260923 17:28:57.320816 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.320304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.193:44089)
0923 17:28:57.320549 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.320645 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260923 17:28:57.324658 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.321698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.8.29.193:36301)
0923 17:28:57.324414 (+ 2716us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.324489 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2623}
I20260923 17:28:57.325233 15074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:57.325460 15074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:28:57.330717 15074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b894fbfff63a4a1297abadee53713db3: Sending DeleteTablet for 3 replicas of tablet c87f2ba6f8c54113b7af9b505f16d8b3
I20260923 17:28:57.331852 15376 tablet_service.cc:1514] Processing DeleteTablet for tablet c87f2ba6f8c54113b7af9b505f16d8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:57 UTC) from {username='oryx', principal='oryx/127.8.29.253@KRBTEST.COM'} at 127.0.0.1:59590
I20260923 17:28:57.331874 15522 tablet_service.cc:1514] Processing DeleteTablet for tablet c87f2ba6f8c54113b7af9b505f16d8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:57 UTC) from {username='oryx', principal='oryx/127.8.29.253@KRBTEST.COM'} at 127.0.0.1:45186
I20260923 17:28:57.331929 15663 tablet_service.cc:1514] Processing DeleteTablet for tablet c87f2ba6f8c54113b7af9b505f16d8b3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:28:57 UTC) from {username='oryx', principal='oryx/127.8.29.253@KRBTEST.COM'} at 127.0.0.1:59990
I20260923 17:28:57.332571 15854 tablet_replica.cc:330] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: stopping tablet replica
I20260923 17:28:57.332783 15854 raft_consensus.cc:2229] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:28:57.332938 15855 tablet_replica.cc:330] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: stopping tablet replica
I20260923 17:28:57.333106 15855 raft_consensus.cc:2229] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:57.333232 15854 raft_consensus.cc:2258] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:57.333375 15855 raft_consensus.cc:2258] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:57.333828 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 15250
I20260923 17:28:57.334338 15854 ts_tablet_manager.cc:1905] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:57.335340 15855 ts_tablet_manager.cc:1905] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:57.338207 15853 tablet_replica.cc:330] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: stopping tablet replica
I20260923 17:28:57.338387 15853 raft_consensus.cc:2229] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:28:57.338675 15853 raft_consensus.cc:2258] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:28:57.339808 15853 ts_tablet_manager.cc:1905] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:28:57.342330 15855 ts_tablet_manager.cc:1918] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:57.342497 15855 log.cc:1196] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/c87f2ba6f8c54113b7af9b505f16d8b3
I20260923 17:28:57.342446 15854 ts_tablet_manager.cc:1918] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:57.342626 15854 log.cc:1196] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/c87f2ba6f8c54113b7af9b505f16d8b3
I20260923 17:28:57.343053 15854 ts_tablet_manager.cc:1939] T c87f2ba6f8c54113b7af9b505f16d8b3 P 455cb603b43346f0b1d56e6dc06f589b: Deleting consensus metadata
I20260923 17:28:57.343743 15855 ts_tablet_manager.cc:1939] T c87f2ba6f8c54113b7af9b505f16d8b3 P b6997baeee22454d9624893c37bca7cf: Deleting consensus metadata
I20260923 17:28:57.344292 15060 catalog_manager.cc:4841] TS 455cb603b43346f0b1d56e6dc06f589b (127.8.29.193:35875): tablet c87f2ba6f8c54113b7af9b505f16d8b3 (table test-table [id=5faed8ffc54a4ab29a327b88752f6588]) successfully deleted
I20260923 17:28:57.345105 15059 catalog_manager.cc:4841] TS b6997baeee22454d9624893c37bca7cf (127.8.29.195:44201): tablet c87f2ba6f8c54113b7af9b505f16d8b3 (table test-table [id=5faed8ffc54a4ab29a327b88752f6588]) successfully deleted
I20260923 17:28:57.346261 15853 ts_tablet_manager.cc:1918] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:28:57.346423 15853 log.cc:1196] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/c87f2ba6f8c54113b7af9b505f16d8b3
I20260923 17:28:57.346872 15853 ts_tablet_manager.cc:1939] T c87f2ba6f8c54113b7af9b505f16d8b3 P d7c0e000eae64596a8d6eb329c4598b9: Deleting consensus metadata
I20260923 17:28:57.347791 15061 catalog_manager.cc:4841] TS d7c0e000eae64596a8d6eb329c4598b9 (127.8.29.194:34613): tablet c87f2ba6f8c54113b7af9b505f16d8b3 (table test-table [id=5faed8ffc54a4ab29a327b88752f6588]) successfully deleted
I20260923 17:28:57.356184 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.335497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.8.29.193:35875)
0923 17:28:57.335664 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:57.335670 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:57.343205 (+ 7535us) server_negotiation.cc:407] Connection header received
0923 17:28:57.343452 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:57.343457 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:57.343541 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:57.343643 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:57.345566 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.346695 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.348139 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.348523 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.348623 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:57.351035 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:57.351051 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:57.351087 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:57.351126 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:57.353707 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:57.354318 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:57.354326 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:57.354328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:57.354446 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:57.354807 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:57.354809 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:57.354811 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:57.355171 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:57.355423 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:57.355674 (+ 251us) server_negotiation.cc:299] Negotiation successful
0923 17:28:57.355854 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:28:57.357002 15436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:57.480115 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 15250
I20260923 17:28:57.499011 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 15453
I20260923 17:28:57.511844 15824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.499752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.8.29.194:34613)
0923 17:28:57.499926 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:57.499930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:57.499987 (+ 57us) server_negotiation.cc:407] Connection header received
0923 17:28:57.500131 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:57.500136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:57.500230 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:57.500314 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:57.501638 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.502323 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.503692 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.504101 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.504203 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:57.506595 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:57.506612 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:57.506647 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:57.506684 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:57.509357 (+ 2673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:57.510037 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:57.510045 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:57.510047 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:57.510192 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:57.510566 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:57.510570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:57.510571 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:57.510930 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:57.511183 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:57.511406 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 17:28:57.511577 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:28:57.512509 15582 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:57.531965 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.531271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52266)
0923 17:28:57.531711 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.531798 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":227,"threads_started":1}
I20260923 17:28:57.533286 15859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.532950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:34369 (local address 127.0.0.1:52280)
0923 17:28:57.533094 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 17:28:57.533180 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:34369: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 17:28:57.605033 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 15453
I20260923 17:28:57.618652 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 15600
I20260923 17:28:57.638504 15861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:57.619440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59998 (local address 127.8.29.195:44201)
0923 17:28:57.619912 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:57.619917 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:57.619933 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:28:57.619985 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:57.619989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:57.620055 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:57.620151 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:57.621471 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.622174 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.623544 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:57.623949 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:57.624062 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:57.632541 (+ 8479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:57.632560 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:57.632585 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:57.632628 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:57.635597 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:57.636390 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:57.636399 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:57.636402 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:57.636536 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:57.636931 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:57.636935 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:57.636938 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:57.637311 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:57.637621 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:57.637917 (+ 296us) server_negotiation.cc:299] Negotiation successful
0923 17:28:57.638172 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":250,"threads_started":1}
I20260923 17:28:57.639258 15723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:57.741055 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 15600
I20260923 17:28:57.755323 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 14937
I20260923 17:28:57.756528 15105 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:57.877784 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 14937
W20260923 17:28:57.894587 15249 connection.cc:535] server connection from 127.0.0.1:51080 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:28:57.894850 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 15038
I20260923 17:28:57.895874 15293 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:28:58.009068 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 15038
2026-09-23T17:28:58Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9408 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:28:58 dist-test-slave-p7jt krb5kdc[15867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:28:58 dist-test-slave-p7jt krb5kdc[15867](info): set up 2 sockets
Sep 23 17:28:58 dist-test-slave-p7jt krb5kdc[15867](info): commencing operation
krb5kdc: starting...
W20260923 17:28:59.027288 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.004s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:28:59 dist-test-slave-p7jt krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:28:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:28:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.254.keytab.
Entry for principal oryx/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.254.keytab.
I20260923 17:28:59.243201 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:35673
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.254.keytab
--principal=oryx/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:28:59.567190 15880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:28:59.567525 15880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:28:59.567651 15880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:28:59.575204 15880 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:28:59.575340 15880 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 17:28:59.575416 15880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:28:59.575506 15880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:28:59.575572 15880 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:28:59.575654 15880 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:28:59.585824 15880 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:35673
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.254.keytab
--principal=oryx/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:28:59.587188 15880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:28:59.588527 15880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:28:59.600049 15888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:59.600319 15886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:28:59.600119 15885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:28:59.601259 15880 server_base.cc:1034] running on GCE node
I20260923 17:28:59.602020 15880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:28:59.603443 15880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:28:59.604682 15880 hybrid_clock.cc:648] HybridClock initialized: now 1790184539604621 us; error 52 us; skew 500 ppm
I20260923 17:28:59.605001 15880 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:28:59 dist-test-slave-p7jt krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184539, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:28:59.608045 15880 init.cc:377] Logged in from keytab as oryx/127.8.29.254@KRBTEST.COM (short username oryx)
I20260923 17:28:59.609498 15880 webserver.cc:460] Webserver started at http://127.8.29.254:43427/ using document root <none> and password file <none>
I20260923 17:28:59.610225 15880 fs_manager.cc:359] Metadata directory not provided
I20260923 17:28:59.610342 15880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:28:59.610755 15880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:28:59.613766 15880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252"
format_stamp: "Formatted at 2026-09-23 17:28:59 on dist-test-slave-p7jt"
server_key: "71dcab0de838c025cbf8c80fecb8a32e"
server_key_iv: "fbdcf9beca9435612082ba51656ea16f"
server_key_version: "encryptionkey@0"
I20260923 17:28:59.614612 15880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252"
format_stamp: "Formatted at 2026-09-23 17:28:59 on dist-test-slave-p7jt"
server_key: "71dcab0de838c025cbf8c80fecb8a32e"
server_key_iv: "fbdcf9beca9435612082ba51656ea16f"
server_key_version: "encryptionkey@0"
I20260923 17:28:59.620133 15880 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:28:59.623850 15895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:28:59.625233 15880 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:28:59.625398 15880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252"
format_stamp: "Formatted at 2026-09-23 17:28:59 on dist-test-slave-p7jt"
server_key: "71dcab0de838c025cbf8c80fecb8a32e"
server_key_iv: "fbdcf9beca9435612082ba51656ea16f"
server_key_version: "encryptionkey@0"
I20260923 17:28:59.625602 15880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:28:59.652424 15880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:28:59.656797 15880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:28:59.657069 15880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:28:59.677568 15880 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:35673
I20260923 17:28:59.677590 15956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:35673 every 8 connection(s)
I20260923 17:28:59.678912 15880 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:28:59.683338 15957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:28:59.683974 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 15880
I20260923 17:28:59.684201 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:28:59.684788 8311 external_mini_cluster.cc:1428] Setting key 5bf68127c212ea0fe1d2e225c6928904
I20260923 17:28:59.697328 15957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252: Bootstrap starting.
Sep 23 17:28:59 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:28:59.701155 15957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252: Neither blocks nor log segments found. Creating new log.
I20260923 17:28:59.702255 15957 log.cc:824] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252: Log is configured to *not* fsync() on all Append() calls
I20260923 17:28:59.705621 15957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252: No bootstrap required, opened a new log
I20260923 17:28:59.708564 15960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:28:59.686818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.8.29.254:35673)
0923 17:28:59.687631 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0923 17:28:59.687643 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:28:59.687695 (+ 52us) server_negotiation.cc:407] Connection header received
0923 17:28:59.689349 (+ 1654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:28:59.689378 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:28:59.689896 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:28:59.690255 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:28:59.692041 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:59.693374 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:59.695146 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:28:59.695870 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:28:59.696028 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:28:59.700696 (+ 4668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:28:59.700706 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:28:59.700722 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:28:59.700768 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:28:59.704092 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:59.705028 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:59.705036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:59.705043 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:59.705265 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:28:59.705761 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:28:59.705767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:28:59.705769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:28:59.706358 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:28:59.706622 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0923 17:28:59.707085 (+ 463us) server_negotiation.cc:299] Negotiation successful
0923 17:28:59.707478 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":363,"threads_started":1}
I20260923 17:28:59.713451 15957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35673 } }
I20260923 17:28:59.713796 15957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:28:59.713891 15957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c5b0da48bc242c6bd3dbce8fa8d2252, State: Initialized, Role: FOLLOWER
I20260923 17:28:59.714412 15957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35673 } }
I20260923 17:28:59.714654 15957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:28:59.714776 15957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:28:59.714920 15957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:28:59.717092 15957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35673 } }
I20260923 17:28:59.717588 15957 leader_election.cc:302] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c5b0da48bc242c6bd3dbce8fa8d2252; no voters:
I20260923 17:28:59.718135 15957 leader_election.cc:288] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:28:59.718261 15962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:28:59.718967 15962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [term 1 LEADER]: Becoming Leader. State: Replica: 2c5b0da48bc242c6bd3dbce8fa8d2252, State: Running, Role: LEADER
I20260923 17:28:59.719563 15962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35673 } }
I20260923 17:28:59.720203 15957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:28:59.722278 15964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c5b0da48bc242c6bd3dbce8fa8d2252. Latest consensus state: current_term: 1 leader_uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35673 } } }
I20260923 17:28:59.722728 15964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:59.723454 15963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c5b0da48bc242c6bd3dbce8fa8d2252" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35673 } } }
I20260923 17:28:59.723722 15963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252 [sys.catalog]: This master's current role is: LEADER
I20260923 17:28:59.724282 15971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:28:59.728963 15971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:28:59.740600 15971 catalog_manager.cc:1347] Generated new cluster ID: 49b5307dce0c4813ba8dcd4942c97297
I20260923 17:28:59.740772 15971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:28:59.761747 15971 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:28:59.763716 15971 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:28:59.773170 15971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252: Generated new TSK 0
I20260923 17:28:59.774302 15971 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab.
Entry for principal oryx/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab.
I20260923 17:28:59.834687 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:35673
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab
--principal=oryx/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:00.158731 15983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:00.159030 15983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:00.159126 15983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:00.166750 15983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:00.166873 15983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:00.166965 15983 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:29:00.167049 15983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:29:00.177021 15983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab
--principal=oryx/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.29.254:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:00.178300 15983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:00.179955 15983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:00.192975 15989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:00.193121 15991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:00.193125 15988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:00.194229 15983 server_base.cc:1034] running on GCE node
I20260923 17:29:00.194926 15983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:00.195915 15983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:00.197201 15983 hybrid_clock.cc:648] HybridClock initialized: now 1790184540197118 us; error 31 us; skew 500 ppm
I20260923 17:29:00.197497 15983 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:00 dist-test-slave-p7jt krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184540, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:00.200564 15983 init.cc:377] Logged in from keytab as oryx/127.8.29.193@KRBTEST.COM (short username oryx)
I20260923 17:29:00.202008 15983 webserver.cc:460] Webserver started at http://127.8.29.193:44015/ using document root <none> and password file <none>
I20260923 17:29:00.202742 15983 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:00.202831 15983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:00.203107 15983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:00.206007 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "a5bb29f2d86245268821153dc10335b8"
format_stamp: "Formatted at 2026-09-23 17:29:00 on dist-test-slave-p7jt"
server_key: "575f208b421e7ddcd9c77baaff54aba7"
server_key_iv: "e018739da40e4528705de444924a8317"
server_key_version: "encryptionkey@0"
I20260923 17:29:00.206826 15983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "a5bb29f2d86245268821153dc10335b8"
format_stamp: "Formatted at 2026-09-23 17:29:00 on dist-test-slave-p7jt"
server_key: "575f208b421e7ddcd9c77baaff54aba7"
server_key_iv: "e018739da40e4528705de444924a8317"
server_key_version: "encryptionkey@0"
I20260923 17:29:00.212123 15983 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:29:00.215809 15998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:00.217281 15983 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:29:00.217432 15983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "a5bb29f2d86245268821153dc10335b8"
format_stamp: "Formatted at 2026-09-23 17:29:00 on dist-test-slave-p7jt"
server_key: "575f208b421e7ddcd9c77baaff54aba7"
server_key_iv: "e018739da40e4528705de444924a8317"
server_key_version: "encryptionkey@0"
I20260923 17:29:00.217653 15983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:00.236457 15983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:00.241101 15983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:00.241335 15983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:00.244243 15983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:00.244367 15983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:00.244521 15983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:00.244599 15983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 17:29:00 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184540, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:29:00.268230 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:00.244714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:41803 (local address 127.8.29.254:35673)
0923 17:29:00.245480 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:00.245486 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:00.246540 (+ 1054us) server_negotiation.cc:407] Connection header received
0923 17:29:00.248722 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:00.248728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:00.248813 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:00.248908 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:00.251264 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.252209 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.255145 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.255685 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.255823 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:00.260959 (+ 5136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:00.260965 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:00.260969 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:00.261005 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:00.264512 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:00.265379 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:00.265385 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:00.265389 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:00.265563 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:00.266144 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:00.266149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:00.266153 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:00.266483 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:00.266644 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:00.267626 (+ 982us) server_negotiation.cc:299] Negotiation successful
0923 17:29:00.267878 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":554,"threads_started":1}
I20260923 17:29:00.269426 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:00.245486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35673 (local address 127.8.29.193:41803)
0923 17:29:00.246350 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:00.246393 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:00.248409 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:00.249212 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:00.249226 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:00.250115 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:00.251056 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:00.251085 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.252397 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.252404 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:00.254966 (+ 2562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:00.254986 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.255979 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.255987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:00.256283 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:00.257042 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:00.257073 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:00.260679 (+ 3606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:00.264711 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:00.264723 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:00.264745 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:00.265237 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:00.265676 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:00.265681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:00.265685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:00.266007 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:00.266613 (+ 606us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:00.266623 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:00.267180 (+ 557us) client_negotiation.cc:769] Sending connection context
0923 17:29:00.267479 (+ 299us) client_negotiation.cc:240] Negotiation successful
0923 17:29:00.268109 (+ 630us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":359,"threads_started":1}
I20260923 17:29:00.292878 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:00.278512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:42725 (local address 127.8.29.254:35673)
0923 17:29:00.278732 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:00.278737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:00.279352 (+ 615us) server_negotiation.cc:407] Connection header received
0923 17:29:00.279527 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:00.279532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:00.279598 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:00.279682 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:00.280654 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.281355 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.283197 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.283702 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.283829 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:00.286245 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:00.286254 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:00.286259 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:00.286305 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:00.289656 (+ 3351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:00.290566 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:00.290574 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:00.290578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:00.290764 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:00.291248 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:00.291254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:00.291257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:00.291678 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:00.291819 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:00.292309 (+ 490us) server_negotiation.cc:299] Negotiation successful
0923 17:29:00.292530 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260923 17:29:00.293021 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:00.278512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35673 (local address 127.8.29.193:42725)
0923 17:29:00.279228 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:00.279249 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:00.279409 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:00.279723 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:00.279728 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:00.280219 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:00.280503 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:00.280513 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.281499 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.281503 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:00.283052 (+ 1549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:00.283068 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.283908 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.283919 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:00.284151 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:00.284955 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:00.284980 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:00.285996 (+ 1016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:00.289797 (+ 3801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:00.289804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:00.289809 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:00.290401 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:00.290921 (+ 520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:00.290927 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:00.290931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:00.291123 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:00.291880 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:00.291886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:00.292048 (+ 162us) client_negotiation.cc:769] Sending connection context
0923 17:29:00.292163 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 17:29:00.292539 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"thread_start_us":413,"threads_started":1}
I20260923 17:29:00.295495 15983 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:42605
I20260923 17:29:00.295512 16118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:42605 every 8 connection(s)
I20260923 17:29:00.296984 15983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:29:00.298964 16119 heartbeater.cc:347] Connected to a master server at 127.8.29.254:35673
I20260923 17:29:00.299177 16119 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:00.299775 16119 heartbeater.cc:511] Master 127.8.29.254:35673 requested a full tablet report, sending...
I20260923 17:29:00.301661 15912 ts_manager.cc:194] Registered new tserver with Master: a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605)
I20260923 17:29:00.303133 15912 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:41803
I20260923 17:29:00.305748 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 15983
I20260923 17:29:00.305948 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:29:00.306581 8311 external_mini_cluster.cc:1428] Setting key 7d750aa1683457f6f3ed5180d57e818d
WARNING: no policy specified for oryx/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab.
Entry for principal oryx/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab.
I20260923 17:29:00.353262 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:35673
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab
--principal=oryx/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:00.678481 16123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:00.678848 16123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:00.679018 16123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:00.690542 16123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:00.690703 16123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:00.690840 16123 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:29:00.690954 16123 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:29:00.704167 16123 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab
--principal=oryx/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.29.254:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:00.705423 16123 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:00.706833 16123 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:00.720324 16128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:00.720345 16131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:00.720285 16129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:00.721201 16123 server_base.cc:1034] running on GCE node
I20260923 17:29:00.721832 16123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:00.723029 16123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:00.724259 16123 hybrid_clock.cc:648] HybridClock initialized: now 1790184540724183 us; error 56 us; skew 500 ppm
I20260923 17:29:00.724592 16123 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:00 dist-test-slave-p7jt krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184540, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:00.727648 16123 init.cc:377] Logged in from keytab as oryx/127.8.29.194@KRBTEST.COM (short username oryx)
I20260923 17:29:00.729118 16123 webserver.cc:460] Webserver started at http://127.8.29.194:34033/ using document root <none> and password file <none>
I20260923 17:29:00.729856 16123 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:00.729955 16123 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:00.730312 16123 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:00.733422 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "749f0bdd7405488998e8a2bcc6c3de95"
format_stamp: "Formatted at 2026-09-23 17:29:00 on dist-test-slave-p7jt"
server_key: "9c193f96813c60e225944aa21166d69a"
server_key_iv: "6e4b6db68398ac6a9b327ca79947683e"
server_key_version: "encryptionkey@0"
I20260923 17:29:00.734256 16123 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "749f0bdd7405488998e8a2bcc6c3de95"
format_stamp: "Formatted at 2026-09-23 17:29:00 on dist-test-slave-p7jt"
server_key: "9c193f96813c60e225944aa21166d69a"
server_key_iv: "6e4b6db68398ac6a9b327ca79947683e"
server_key_version: "encryptionkey@0"
I20260923 17:29:00.739559 16123 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:29:00.743268 16138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:00.744627 16123 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:29:00.744786 16123 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "749f0bdd7405488998e8a2bcc6c3de95"
format_stamp: "Formatted at 2026-09-23 17:29:00 on dist-test-slave-p7jt"
server_key: "9c193f96813c60e225944aa21166d69a"
server_key_iv: "6e4b6db68398ac6a9b327ca79947683e"
server_key_version: "encryptionkey@0"
I20260923 17:29:00.745002 16123 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:00.769192 16123 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:00.774067 16123 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:00.774335 16123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:00.777153 16123 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:00.777256 16123 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:00.777413 16123 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:00.777552 16123 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 23 17:29:00 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184540, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:29:00.801919 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:00.777579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:52235 (local address 127.8.29.254:35673)
0923 17:29:00.777729 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:00.777733 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:00.779365 (+ 1632us) server_negotiation.cc:407] Connection header received
0923 17:29:00.781410 (+ 2045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:00.781415 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:00.781502 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:00.781588 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:00.783894 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.785074 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.787410 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.788113 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.788271 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:00.794533 (+ 6262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:00.794542 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:00.794547 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:00.794603 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:00.797983 (+ 3380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:00.799167 (+ 1184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:00.799174 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:00.799177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:00.799334 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:00.799994 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:00.799999 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:00.800003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:00.800458 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:00.800599 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:00.801358 (+ 759us) server_negotiation.cc:299] Negotiation successful
0923 17:29:00.801583 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:29:00.803148 16147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:00.778363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35673 (local address 127.8.29.194:52235)
0923 17:29:00.779203 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:00.779249 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:00.781123 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:00.781907 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:00.781922 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:00.782746 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:00.783622 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:00.783648 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.785252 (+ 1604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.785258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:00.787198 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:00.787211 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.789051 (+ 1840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.789060 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:00.789412 (+ 352us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:00.790314 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:00.790352 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:00.794166 (+ 3814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:00.798198 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:00.798210 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:00.798234 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:00.798962 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:00.799493 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:00.799499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:00.799502 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:00.799871 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:00.800599 (+ 728us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:00.800607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:00.800990 (+ 383us) client_negotiation.cc:769] Sending connection context
0923 17:29:00.801234 (+ 244us) client_negotiation.cc:240] Negotiation successful
0923 17:29:00.801795 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":348,"threads_started":1}
I20260923 17:29:00.823278 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:00.808567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:40039 (local address 127.8.29.254:35673)
0923 17:29:00.808762 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:00.808769 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:00.809296 (+ 527us) server_negotiation.cc:407] Connection header received
0923 17:29:00.809488 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:00.809493 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:00.809584 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:00.809676 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:00.810973 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.812084 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.814638 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.815034 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.815136 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:00.817636 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:00.817644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:00.817648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:00.817689 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:00.820417 (+ 2728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:00.821327 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:00.821332 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:00.821336 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:00.821507 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:00.821905 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:00.821908 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:00.821910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:00.822241 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:00.822370 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:00.822855 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 17:29:00.823020 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:29:00.823657 16213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:00.808536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35673 (local address 127.8.29.194:40039)
0923 17:29:00.809192 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:00.809212 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:00.809368 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:00.809733 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:00.809737 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:00.810330 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:00.810804 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:00.810822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.812317 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.812322 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:00.814486 (+ 2164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:00.814506 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:00.815230 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:00.815237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:00.815403 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:00.816076 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:00.816099 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:00.817345 (+ 1246us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:00.820545 (+ 3200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:00.820551 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:00.820555 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:00.821200 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:00.821634 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:00.821639 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:00.821643 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:00.821806 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:00.822337 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:00.822343 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:00.822613 (+ 270us) client_negotiation.cc:769] Sending connection context
0923 17:29:00.822761 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 17:29:00.823134 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":321,"threads_started":1}
I20260923 17:29:00.829723 16123 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:34677
I20260923 17:29:00.829743 16257 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:34677 every 8 connection(s)
I20260923 17:29:00.830957 16123 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:29:00.832985 16258 heartbeater.cc:347] Connected to a master server at 127.8.29.254:35673
I20260923 17:29:00.833217 16258 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:00.833828 16258 heartbeater.cc:511] Master 127.8.29.254:35673 requested a full tablet report, sending...
I20260923 17:29:00.834504 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 16123
I20260923 17:29:00.834707 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:29:00.835155 8311 external_mini_cluster.cc:1428] Setting key b63315bcab164ac80fbe60883b4cfcb0
I20260923 17:29:00.835289 15912 ts_manager.cc:194] Registered new tserver with Master: 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677)
I20260923 17:29:00.836261 15912 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:52235
WARNING: no policy specified for oryx/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab.
Entry for principal oryx/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab.
I20260923 17:29:00.882411 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:35673
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab
--principal=oryx/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:40105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:01.204000 16262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:01.204280 16262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:01.204398 16262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:01.211813 16262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:01.211928 16262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:01.212056 16262 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 17:29:01.212158 16262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:29:01.222385 16262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:40105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab
--principal=oryx/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.29.254:35673
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:01.223783 16262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:01.225152 16262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:01.238538 16268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:01.238989 16267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:01.239862 16270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:01.240648 16262 server_base.cc:1034] running on GCE node
I20260923 17:29:01.241323 16262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:01.242360 16262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:01.243613 16262 hybrid_clock.cc:648] HybridClock initialized: now 1790184541243545 us; error 39 us; skew 500 ppm
I20260923 17:29:01.243948 16262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:01 dist-test-slave-p7jt krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184541, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:01.247181 16262 init.cc:377] Logged in from keytab as oryx/127.8.29.195@KRBTEST.COM (short username oryx)
I20260923 17:29:01.248678 16262 webserver.cc:460] Webserver started at http://127.8.29.195:34469/ using document root <none> and password file <none>
I20260923 17:29:01.249379 16262 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:01.249495 16262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:01.249835 16262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:01.252774 16262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "93c0838c331e4d16b93b5991368190fc"
format_stamp: "Formatted at 2026-09-23 17:29:01 on dist-test-slave-p7jt"
server_key: "0d681b02d670a27e6865dfdf2dccbfbb"
server_key_iv: "fc51b7da02619980a07b793af5708a9f"
server_key_version: "encryptionkey@0"
I20260923 17:29:01.253526 16262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "93c0838c331e4d16b93b5991368190fc"
format_stamp: "Formatted at 2026-09-23 17:29:01 on dist-test-slave-p7jt"
server_key: "0d681b02d670a27e6865dfdf2dccbfbb"
server_key_iv: "fc51b7da02619980a07b793af5708a9f"
server_key_version: "encryptionkey@0"
I20260923 17:29:01.258579 16262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:29:01.262097 16277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:01.263720 16262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:29:01.263870 16262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "93c0838c331e4d16b93b5991368190fc"
format_stamp: "Formatted at 2026-09-23 17:29:01 on dist-test-slave-p7jt"
server_key: "0d681b02d670a27e6865dfdf2dccbfbb"
server_key_iv: "fc51b7da02619980a07b793af5708a9f"
server_key_version: "encryptionkey@0"
I20260923 17:29:01.264067 16262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:01.284767 16262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:01.289263 16262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:01.289496 16262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:01.292193 16262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:01.292276 16262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:01.292407 16262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:01.292456 16262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:01.305981 16119 heartbeater.cc:503] Master 127.8.29.254:35673 was elected leader, sending a full tablet report...
Sep 23 17:29:01 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184541, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:29:01.316452 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.293015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:41597 (local address 127.8.29.254:35673)
0923 17:29:01.295201 (+ 2186us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.295207 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.295232 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:29:01.296910 (+ 1678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.296916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.296999 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.297084 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:01.299424 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.300447 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.303594 (+ 3147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.304059 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.304170 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.309378 (+ 5208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:01.309384 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:01.309387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:01.309421 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:01.312418 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:01.313555 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:01.313562 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:01.313565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:01.313739 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:01.314189 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:01.314195 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:01.314199 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:01.314597 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:01.314744 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.315833 (+ 1089us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.316096 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2049}
I20260923 17:29:01.317529 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.293538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35673 (local address 127.8.29.195:41597)
0923 17:29:01.294487 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:01.294544 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:01.296635 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:01.297362 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:01.297377 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:01.298175 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:01.299169 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.299200 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.300669 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.300675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.303341 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.303362 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.304322 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.304330 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.304661 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.305619 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:01.305663 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:01.309095 (+ 3432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:01.312567 (+ 3472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:01.312581 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:01.312602 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:01.313356 (+ 754us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:01.313857 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:01.313861 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:01.313864 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:01.314055 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:01.314744 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:01.314754 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:01.315340 (+ 586us) client_negotiation.cc:769] Sending connection context
0923 17:29:01.315696 (+ 356us) client_negotiation.cc:240] Negotiation successful
0923 17:29:01.316274 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":412,"threads_started":1}
I20260923 17:29:01.337896 16348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.323891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35673 (local address 127.8.29.195:60245)
0923 17:29:01.324551 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:01.324573 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:01.324751 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:01.325085 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:01.325090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:01.325616 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:01.325902 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.325911 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.327005 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.327012 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.328899 (+ 1887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.328919 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.329679 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.329686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.329866 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.330547 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:01.330573 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:01.331853 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:01.334954 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:01.334959 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:01.334967 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:01.335661 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:01.336086 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:01.336091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:01.336098 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:01.336272 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:01.336979 (+ 707us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:01.336983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:01.337165 (+ 182us) client_negotiation.cc:769] Sending connection context
0923 17:29:01.337281 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 17:29:01.337544 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":325,"threads_started":1}
I20260923 17:29:01.338014 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.323895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:60245 (local address 127.8.29.254:35673)
0923 17:29:01.324064 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.324080 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.324685 (+ 605us) server_negotiation.cc:407] Connection header received
0923 17:29:01.324872 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.324876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.324939 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.325029 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:01.326048 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.326846 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.329087 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.329484 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.329588 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.332115 (+ 2527us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:01.332121 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:01.332125 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:01.332161 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:01.334775 (+ 2614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:01.335800 (+ 1025us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:01.335806 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:01.335810 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:01.335956 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:01.336429 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:01.336435 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:01.336437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:01.336774 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:01.336924 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.337423 (+ 499us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.337680 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:29:01.348822 16396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:33765 every 8 connection(s)
I20260923 17:29:01.348848 16262 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:33765
I20260923 17:29:01.350127 16262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:29:01.352250 16397 heartbeater.cc:347] Connected to a master server at 127.8.29.254:35673
I20260923 17:29:01.352450 16397 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:01.353119 16397 heartbeater.cc:511] Master 127.8.29.254:35673 requested a full tablet report, sending...
I20260923 17:29:01.353150 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 16262
I20260923 17:29:01.353314 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:29:01.353703 8311 external_mini_cluster.cc:1428] Setting key 27423128fc5a8854424ff5f507e69591
I20260923 17:29:01.354707 15911 ts_manager.cc:194] Registered new tserver with Master: 93c0838c331e4d16b93b5991368190fc (127.8.29.195:33765)
I20260923 17:29:01.355703 15911 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:41597
I20260923 17:29:01.356667 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 17:29:01 dist-test-slave-p7jt krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 17:29:01 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:01 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:29:01.403748 16007 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:01.393831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51532 (local address 127.8.29.254:35673)
0923 17:29:01.394065 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.394071 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.394230 (+ 159us) server_negotiation.cc:407] Connection header received
0923 17:29:01.394414 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.394419 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.394528 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.394624 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:01.395957 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.396685 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.397898 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.398312 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.398424 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.403468 (+ 5044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51532: BlockingRecv error: recv got EOF from 127.0.0.1:51532 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 17:29:01 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184541, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:29:01.420722 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.407374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51544 (local address 127.8.29.254:35673)
0923 17:29:01.407566 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.407573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.407841 (+ 268us) server_negotiation.cc:407] Connection header received
0923 17:29:01.408027 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.408033 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.408114 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.408198 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:01.409338 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.410071 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.411350 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.411757 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.411870 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.415444 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:01.415450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:01.415454 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:01.415490 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:01.418036 (+ 2546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:01.418780 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:01.418784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:01.418787 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:01.418919 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:01.419315 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:01.419319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:01.419321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:01.419622 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:01.419855 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.420209 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.420391 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:29:01.424700 15912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:29:01.431671 15912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:29:01.455791 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.444005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:33765 (local address 127.0.0.1:38942)
0923 17:29:01.445030 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:01.445083 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:01.445274 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:01.447059 (+ 1785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:01.447069 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:01.447104 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:01.447542 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.447556 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.449926 (+ 2370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.449933 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.452595 (+ 2662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.452615 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.454809 (+ 2194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.454815 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.455034 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.455086 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 17:29:01.455218 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 17:29:01.455416 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":723,"thread_start_us":8167,"threads_started":1}
I20260923 17:29:01.456754 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.444168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.8.29.195:33765)
0923 17:29:01.445992 (+ 1824us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.446003 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.446037 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:29:01.446141 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.446150 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.446446 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.446685 (+ 239us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:01.447818 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.449735 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.452821 (+ 3086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.454269 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.454376 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.454664 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.456260 (+ 1596us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.456435 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1634,"thread_start_us":1442,"threads_started":1}
I20260923 17:29:01.459949 16415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.444191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34677 (local address 127.0.0.1:42888)
0923 17:29:01.452529 (+ 8338us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:01.452554 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:01.452705 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:01.453702 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:01.453707 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:01.453734 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:01.454166 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.454179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.455890 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.455894 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.457792 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.457804 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.459232 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.459237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.459399 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.459430 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:29:01.459512 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:29:01.459632 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8168,"thread_start_us":752,"threads_started":1}
I20260923 17:29:01.460623 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.444757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.8.29.194:34677)
0923 17:29:01.452844 (+ 8087us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.452854 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.452883 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:29:01.452978 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.452986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.453296 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.453497 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:01.454360 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.455736 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.457946 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.459038 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.459159 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.459381 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.460172 (+ 791us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.460347 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7918,"thread_start_us":7772,"threads_started":1}
I20260923 17:29:01.461491 16327 tablet_service.cc:1467] Processing CreateTablet for tablet bd4211886f6d4caf8ea3b40f0fce8225 (DEFAULT_TABLE table=test-table [id=7468f14064c0415bb3b14cd4beb0c6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:01.463037 16327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4211886f6d4caf8ea3b40f0fce8225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:01.463899 16188 tablet_service.cc:1467] Processing CreateTablet for tablet bd4211886f6d4caf8ea3b40f0fce8225 (DEFAULT_TABLE table=test-table [id=7468f14064c0415bb3b14cd4beb0c6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:01.464746 16416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.445998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42605 (local address 127.0.0.1:52480)
0923 17:29:01.457068 (+ 11070us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:01.457088 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:01.457188 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:01.458371 (+ 1183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:01.458377 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:01.458404 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:01.458574 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.458585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.460322 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.460327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.462040 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.462055 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.464059 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.464065 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.464244 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.464277 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:29:01.464360 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 17:29:01.464497 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10969,"thread_start_us":10776,"threads_started":1}
I20260923 17:29:01.465143 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.446231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.8.29.193:42605)
0923 17:29:01.457781 (+ 11550us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.457789 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.457818 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:29:01.457907 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.457915 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.458175 (+ 260us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.458378 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:01.458716 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.460135 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.462244 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.463641 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.463764 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.463962 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.464688 (+ 726us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.464867 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11382,"thread_start_us":11670,"threads_started":1}
I20260923 17:29:01.465291 16188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4211886f6d4caf8ea3b40f0fce8225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:01.468817 16049 tablet_service.cc:1467] Processing CreateTablet for tablet bd4211886f6d4caf8ea3b40f0fce8225 (DEFAULT_TABLE table=test-table [id=7468f14064c0415bb3b14cd4beb0c6d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:01.470287 16049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd4211886f6d4caf8ea3b40f0fce8225. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:01.476945 16421 tablet_bootstrap.cc:492] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: Bootstrap starting.
I20260923 17:29:01.480113 16422 tablet_bootstrap.cc:492] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: Bootstrap starting.
I20260923 17:29:01.481212 16421 tablet_bootstrap.cc:654] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:01.482550 16421 log.cc:824] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:01.483551 16422 tablet_bootstrap.cc:654] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:01.484655 16424 tablet_bootstrap.cc:492] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: Bootstrap starting.
I20260923 17:29:01.484752 16422 log.cc:824] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:01.486285 16421 tablet_bootstrap.cc:492] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: No bootstrap required, opened a new log
I20260923 17:29:01.486737 16421 ts_tablet_manager.cc:1397] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260923 17:29:01.487983 16424 tablet_bootstrap.cc:654] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:01.488932 16424 log.cc:824] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:01.491786 16422 tablet_bootstrap.cc:492] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: No bootstrap required, opened a new log
I20260923 17:29:01.491990 16424 tablet_bootstrap.cc:492] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: No bootstrap required, opened a new log
I20260923 17:29:01.492051 16422 ts_tablet_manager.cc:1397] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260923 17:29:01.492265 16424 ts_tablet_manager.cc:1397] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20260923 17:29:01.497179 16421 raft_consensus.cc:348] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } }
I20260923 17:29:01.497686 16421 raft_consensus.cc:374] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:01.497835 16421 raft_consensus.cc:729] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749f0bdd7405488998e8a2bcc6c3de95, State: Initialized, Role: FOLLOWER
I20260923 17:29:01.498543 16421 consensus_queue.cc:260] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } }
I20260923 17:29:01.499691 16258 heartbeater.cc:503] Master 127.8.29.254:35673 was elected leader, sending a full tablet report...
I20260923 17:29:01.500136 16422 raft_consensus.cc:348] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } }
I20260923 17:29:01.500495 16422 raft_consensus.cc:374] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:01.500591 16422 raft_consensus.cc:729] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c0838c331e4d16b93b5991368190fc, State: Initialized, Role: FOLLOWER
I20260923 17:29:01.501199 16422 consensus_queue.cc:260] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } }
I20260923 17:29:01.502089 16421 ts_tablet_manager.cc:1428] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260923 17:29:01.502125 16424 raft_consensus.cc:348] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } }
I20260923 17:29:01.502655 16424 raft_consensus.cc:374] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:01.503123 16397 heartbeater.cc:503] Master 127.8.29.254:35673 was elected leader, sending a full tablet report...
I20260923 17:29:01.503818 16422 ts_tablet_manager.cc:1428] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: Time spent starting tablet: real 0.012s user 0.009s sys 0.001s
I20260923 17:29:01.502888 16424 raft_consensus.cc:729] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bb29f2d86245268821153dc10335b8, State: Initialized, Role: FOLLOWER
I20260923 17:29:01.504703 16424 consensus_queue.cc:260] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } }
I20260923 17:29:01.506765 16424 ts_tablet_manager.cc:1428] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260923 17:29:01.545611 16428 raft_consensus.cc:482] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:29:01.545930 16428 raft_consensus.cc:504] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } }
I20260923 17:29:01.547731 16428 leader_election.cc:288] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677), a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605)
W20260923 17:29:01.549276 16120 tablet.cc:2369] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:29:01.554056 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.548202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34677 (local address 127.8.29.195:60383)
0923 17:29:01.548422 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:01.548439 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:01.548580 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:01.549098 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:01.549103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:01.549134 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:01.549309 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.549318 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.550717 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.550721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.552340 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.552355 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.553483 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.553488 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.553642 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.553667 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:29:01.553738 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:29:01.553841 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65,"mutex_wait_us":4}
I20260923 17:29:01.554222 16418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.548265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:60383 (local address 127.8.29.194:34677)
0923 17:29:01.548470 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.548476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.548498 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:29:01.548752 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.548758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.548818 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.548903 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:01.549449 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.550582 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.552504 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.553328 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.553496 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.553576 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.553793 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.553931 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:29:01.555419 16420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.548786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:58485 (local address 127.8.29.193:42605)
0923 17:29:01.549008 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.549014 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.549488 (+ 474us) server_negotiation.cc:407] Connection header received
0923 17:29:01.549744 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.549750 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.549855 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.549939 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:01.550662 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.551388 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.553337 (+ 1949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.554430 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.554652 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.554705 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.555058 (+ 353us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.555187 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:29:01.555521 16430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.548608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42605 (local address 127.8.29.195:58485)
0923 17:29:01.549386 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:01.549409 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:01.549590 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:01.550101 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:01.550106 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:01.550130 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:01.550476 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.550488 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.551528 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.551534 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.553167 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:01.553179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.554713 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.554719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:01.554883 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.554915 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:29:01.555015 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 17:29:01.555165 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":1114,"threads_started":1}
I20260923 17:29:01.555567 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4211886f6d4caf8ea3b40f0fce8225" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f0bdd7405488998e8a2bcc6c3de95" is_pre_election: true
I20260923 17:29:01.556190 16208 raft_consensus.cc:2454] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 0.
I20260923 17:29:01.556799 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4211886f6d4caf8ea3b40f0fce8225" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bb29f2d86245268821153dc10335b8" is_pre_election: true
I20260923 17:29:01.557008 16278 leader_election.cc:302] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 749f0bdd7405488998e8a2bcc6c3de95, 93c0838c331e4d16b93b5991368190fc; no voters:
I20260923 17:29:01.557256 16069 raft_consensus.cc:2454] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 0.
I20260923 17:29:01.557435 16428 raft_consensus.cc:2781] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:29:01.557569 16428 raft_consensus.cc:482] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:29:01.557682 16428 raft_consensus.cc:3037] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:01.559983 16428 raft_consensus.cc:504] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } }
I20260923 17:29:01.560739 16428 leader_election.cc:288] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 election: Requested vote from peers 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677), a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605)
I20260923 17:29:01.561286 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4211886f6d4caf8ea3b40f0fce8225" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bb29f2d86245268821153dc10335b8"
I20260923 17:29:01.561297 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd4211886f6d4caf8ea3b40f0fce8225" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f0bdd7405488998e8a2bcc6c3de95"
I20260923 17:29:01.561528 16208 raft_consensus.cc:3037] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:01.561528 16069 raft_consensus.cc:3037] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:01.564756 16208 raft_consensus.cc:2454] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 1.
I20260923 17:29:01.564754 16069 raft_consensus.cc:2454] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 1.
I20260923 17:29:01.565251 16278 leader_election.cc:302] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 749f0bdd7405488998e8a2bcc6c3de95, 93c0838c331e4d16b93b5991368190fc; no voters:
I20260923 17:29:01.565591 16428 raft_consensus.cc:2781] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:01.566045 16428 raft_consensus.cc:686] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 1 LEADER]: Becoming Leader. State: Replica: 93c0838c331e4d16b93b5991368190fc, State: Running, Role: LEADER
I20260923 17:29:01.566567 16428 consensus_queue.cc:237] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } }
I20260923 17:29:01.571633 15912 catalog_manager.cc:5495] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 93c0838c331e4d16b93b5991368190fc (127.8.29.195). New cstate: current_term: 1 leader_uuid: "93c0838c331e4d16b93b5991368190fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } health_report { overall_health: UNKNOWN } } }
W20260923 17:29:01.582892 16259 tablet.cc:2369] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:29:01.592720 16417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:01.587485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.8.29.195:33765)
0923 17:29:01.587725 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:01.587730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:01.587749 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:29:01.587859 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:01.587863 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:01.587952 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:01.588077 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:01.588447 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.589182 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.590593 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:01.591065 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:01.591177 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:01.591608 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:01.592144 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:01.592249 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:01.592358 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 17:29:01.592431 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260923 17:29:01.601652 16398 tablet.cc:2369] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:29:01.601799 16069 raft_consensus.cc:1261] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Refusing update from remote peer 93c0838c331e4d16b93b5991368190fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:01.601792 16208 raft_consensus.cc:1261] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Refusing update from remote peer 93c0838c331e4d16b93b5991368190fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:01.602856 16431 consensus_queue.cc:1035] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:01.603184 16428 consensus_queue.cc:1035] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:01.614589 16434 mvcc.cc:205] Tried to move back new op lower bound from 7332595882389598208 to 7332595882260451328. Current Snapshot: MvccSnapshot[applied={T|T < 7332595882389598208}]
I20260923 17:29:01.615813 16435 mvcc.cc:205] Tried to move back new op lower bound from 7332595882389598208 to 7332595882260451328. Current Snapshot: MvccSnapshot[applied={T|T < 7332595882389598208}]
I20260923 17:29:01.628258 15910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:01.628607 15910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:01.632592 15910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252: Sending DeleteTablet for 3 replicas of tablet bd4211886f6d4caf8ea3b40f0fce8225
I20260923 17:29:01.633734 16327 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4211886f6d4caf8ea3b40f0fce8225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:01 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:38942
I20260923 17:29:01.634275 16188 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4211886f6d4caf8ea3b40f0fce8225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:01 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:42888
I20260923 17:29:01.634642 16444 tablet_replica.cc:330] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: stopping tablet replica
I20260923 17:29:01.635074 16444 raft_consensus.cc:2229] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:29:01.635187 16445 tablet_replica.cc:330] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: stopping tablet replica
I20260923 17:29:01.635699 16445 raft_consensus.cc:2229] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:01.635779 16444 raft_consensus.cc:2258] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:01.635749 16049 tablet_service.cc:1514] Processing DeleteTablet for tablet bd4211886f6d4caf8ea3b40f0fce8225 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:01 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:52480
I20260923 17:29:01.636519 16445 raft_consensus.cc:2258] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:01.637880 16444 ts_tablet_manager.cc:1905] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:01.638602 16445 ts_tablet_manager.cc:1905] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:01.638870 15912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:29:01.639556 15912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:29:01.645430 16445 ts_tablet_manager.cc:1918] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:29:01.646124 16446 tablet_replica.cc:330] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: stopping tablet replica
I20260923 17:29:01.646700 16446 raft_consensus.cc:2229] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:01.647231 16446 raft_consensus.cc:2258] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:01.647643 16445 log.cc:1196] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/bd4211886f6d4caf8ea3b40f0fce8225
I20260923 17:29:01.648061 16327 tablet_service.cc:1467] Processing CreateTablet for tablet d1c7861b90bc4d898a924f8769f2c2f4 (DEFAULT_TABLE table=test-table [id=fa4ff00212d64cae98d5ebbfa3e1cf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:01.648530 16327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c7861b90bc4d898a924f8769f2c2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:01.648682 16445 ts_tablet_manager.cc:1939] T bd4211886f6d4caf8ea3b40f0fce8225 P 749f0bdd7405488998e8a2bcc6c3de95: Deleting consensus metadata
I20260923 17:29:01.649911 16446 ts_tablet_manager.cc:1905] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:01.650929 15896 catalog_manager.cc:4841] TS 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677): tablet bd4211886f6d4caf8ea3b40f0fce8225 (table test-table [id=7468f14064c0415bb3b14cd4beb0c6d7]) successfully deleted
I20260923 17:29:01.651446 16188 tablet_service.cc:1467] Processing CreateTablet for tablet d1c7861b90bc4d898a924f8769f2c2f4 (DEFAULT_TABLE table=test-table [id=fa4ff00212d64cae98d5ebbfa3e1cf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:01.651857 16188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c7861b90bc4d898a924f8769f2c2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:01.655783 16444 ts_tablet_manager.cc:1918] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:29:01.655964 16444 log.cc:1196] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/bd4211886f6d4caf8ea3b40f0fce8225
I20260923 17:29:01.656246 16422 tablet_bootstrap.cc:492] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: Bootstrap starting.
I20260923 17:29:01.656503 16444 ts_tablet_manager.cc:1939] T bd4211886f6d4caf8ea3b40f0fce8225 P 93c0838c331e4d16b93b5991368190fc: Deleting consensus metadata
I20260923 17:29:01.657886 16422 tablet_bootstrap.cc:654] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:01.658238 16049 tablet_service.cc:1467] Processing CreateTablet for tablet d1c7861b90bc4d898a924f8769f2c2f4 (DEFAULT_TABLE table=test-table [id=fa4ff00212d64cae98d5ebbfa3e1cf8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:01.658339 15897 catalog_manager.cc:4841] TS 93c0838c331e4d16b93b5991368190fc (127.8.29.195:33765): tablet bd4211886f6d4caf8ea3b40f0fce8225 (table test-table [id=7468f14064c0415bb3b14cd4beb0c6d7]) successfully deleted
I20260923 17:29:01.659412 16421 tablet_bootstrap.cc:492] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: Bootstrap starting.
I20260923 17:29:01.660688 16422 tablet_bootstrap.cc:492] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: No bootstrap required, opened a new log
I20260923 17:29:01.660846 16422 ts_tablet_manager.cc:1397] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260923 17:29:01.661046 16421 tablet_bootstrap.cc:654] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:01.662413 16049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d1c7861b90bc4d898a924f8769f2c2f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:01.663151 16422 raft_consensus.cc:348] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:01.663384 16422 raft_consensus.cc:374] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:01.663486 16422 raft_consensus.cc:729] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c0838c331e4d16b93b5991368190fc, State: Initialized, Role: FOLLOWER
I20260923 17:29:01.663812 16422 consensus_queue.cc:260] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:01.664502 16422 ts_tablet_manager.cc:1428] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:29:01.666716 16446 ts_tablet_manager.cc:1918] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:29:01.666929 16446 log.cc:1196] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/bd4211886f6d4caf8ea3b40f0fce8225
I20260923 17:29:01.667539 16446 ts_tablet_manager.cc:1939] T bd4211886f6d4caf8ea3b40f0fce8225 P a5bb29f2d86245268821153dc10335b8: Deleting consensus metadata
I20260923 17:29:01.670012 16421 tablet_bootstrap.cc:492] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: No bootstrap required, opened a new log
I20260923 17:29:01.670336 16421 ts_tablet_manager.cc:1397] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260923 17:29:01.670329 15896 catalog_manager.cc:4841] TS a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605): tablet bd4211886f6d4caf8ea3b40f0fce8225 (table test-table [id=7468f14064c0415bb3b14cd4beb0c6d7]) successfully deleted
I20260923 17:29:01.671217 16421 raft_consensus.cc:348] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:01.671475 16421 raft_consensus.cc:374] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:01.671576 16421 raft_consensus.cc:729] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749f0bdd7405488998e8a2bcc6c3de95, State: Initialized, Role: FOLLOWER
I20260923 17:29:01.671832 16421 consensus_queue.cc:260] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:01.674041 16424 tablet_bootstrap.cc:492] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: Bootstrap starting.
I20260923 17:29:01.674144 16421 ts_tablet_manager.cc:1428] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:29:01.675683 16424 tablet_bootstrap.cc:654] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:01.677721 16424 tablet_bootstrap.cc:492] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: No bootstrap required, opened a new log
I20260923 17:29:01.677867 16424 ts_tablet_manager.cc:1397] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260923 17:29:01.678685 16424 raft_consensus.cc:348] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:01.678915 16424 raft_consensus.cc:374] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:01.679018 16424 raft_consensus.cc:729] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bb29f2d86245268821153dc10335b8, State: Initialized, Role: FOLLOWER
I20260923 17:29:01.679324 16424 consensus_queue.cc:260] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:01.680763 16424 ts_tablet_manager.cc:1428] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260923 17:29:02.004098 16439 raft_consensus.cc:482] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:29:02.004369 16439 raft_consensus.cc:504] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.005195 16439 leader_election.cc:288] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605), 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677)
I20260923 17:29:02.005909 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c7861b90bc4d898a924f8769f2c2f4" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f0bdd7405488998e8a2bcc6c3de95" is_pre_election: true
I20260923 17:29:02.005908 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c7861b90bc4d898a924f8769f2c2f4" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bb29f2d86245268821153dc10335b8" is_pre_election: true
I20260923 17:29:02.006170 16069 raft_consensus.cc:2454] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 0.
I20260923 17:29:02.006168 16208 raft_consensus.cc:2454] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 0.
I20260923 17:29:02.006649 16278 leader_election.cc:302] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93c0838c331e4d16b93b5991368190fc, a5bb29f2d86245268821153dc10335b8; no voters:
I20260923 17:29:02.006927 16439 raft_consensus.cc:2781] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:29:02.007067 16439 raft_consensus.cc:482] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:29:02.007169 16439 raft_consensus.cc:3037] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.010250 16439 raft_consensus.cc:504] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.010906 16439 leader_election.cc:288] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 election: Requested vote from peers a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605), 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677)
I20260923 17:29:02.011495 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c7861b90bc4d898a924f8769f2c2f4" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f0bdd7405488998e8a2bcc6c3de95"
I20260923 17:29:02.011489 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d1c7861b90bc4d898a924f8769f2c2f4" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bb29f2d86245268821153dc10335b8"
I20260923 17:29:02.011716 16208 raft_consensus.cc:3037] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.011716 16069 raft_consensus.cc:3037] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.014642 16069 raft_consensus.cc:2454] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 1.
I20260923 17:29:02.014642 16208 raft_consensus.cc:2454] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 1.
I20260923 17:29:02.015115 16278 leader_election.cc:302] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93c0838c331e4d16b93b5991368190fc, a5bb29f2d86245268821153dc10335b8; no voters:
I20260923 17:29:02.015448 16439 raft_consensus.cc:2781] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:02.015617 16439 raft_consensus.cc:686] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 1 LEADER]: Becoming Leader. State: Replica: 93c0838c331e4d16b93b5991368190fc, State: Running, Role: LEADER
I20260923 17:29:02.015900 16439 consensus_queue.cc:237] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.018651 15910 catalog_manager.cc:5495] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 93c0838c331e4d16b93b5991368190fc (127.8.29.195). New cstate: current_term: 1 leader_uuid: "93c0838c331e4d16b93b5991368190fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } health_report { overall_health: HEALTHY } } }
Sep 23 17:29:02 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184539, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.254@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:29:02.062548 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.048857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35673 (local address 127.0.0.1:51556)
0923 17:29:02.049674 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.049690 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.049822 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.050151 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.050155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.051040 (+ 885us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:02.051266 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.051276 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.052748 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.052752 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.053703 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.053715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.054396 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.054401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.054609 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.055228 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:02.055245 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:02.057301 (+ 2056us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:02.060310 (+ 3009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:02.060327 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:02.060333 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:02.060777 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:02.061096 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:02.061100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:02.061102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:02.061247 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:02.061710 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:02.061716 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:02.061862 (+ 146us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.061998 (+ 136us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.062213 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":512,"threads_started":1}
I20260923 17:29:02.062657 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.048857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51556 (local address 127.8.29.254:35673)
0923 17:29:02.049539 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.049545 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.049770 (+ 225us) server_negotiation.cc:407] Connection header received
0923 17:29:02.049922 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.049927 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.050001 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.050118 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:02.051440 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.052592 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.053825 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.054253 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.054388 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.057500 (+ 3112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:02.057506 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:02.057509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:02.057541 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:02.060162 (+ 2621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.060885 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.060889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.060891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.061005 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.061330 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.061333 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.061336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.061601 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:02.061703 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.062071 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.062328 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":83,"server-negotiator.queue_time_us":524,"thread_start_us":271,"threads_started":1}
I20260923 17:29:02.067603 15910 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:51556:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 17:29:02.068682 15910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:29:02.075559 16188 tablet_service.cc:1467] Processing CreateTablet for tablet eca0c6b3346f4eaaae07966629353c3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72c5f2a52b244712b2223b8fa29aac35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:29:02.075559 16049 tablet_service.cc:1467] Processing CreateTablet for tablet eca0c6b3346f4eaaae07966629353c3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72c5f2a52b244712b2223b8fa29aac35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:29:02.075726 16327 tablet_service.cc:1467] Processing CreateTablet for tablet eca0c6b3346f4eaaae07966629353c3b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=72c5f2a52b244712b2223b8fa29aac35]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 17:29:02.076143 16188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca0c6b3346f4eaaae07966629353c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.076143 16049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca0c6b3346f4eaaae07966629353c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.076231 16327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eca0c6b3346f4eaaae07966629353c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.081460 16422 tablet_bootstrap.cc:492] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: Bootstrap starting.
I20260923 17:29:02.082873 16422 tablet_bootstrap.cc:654] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.083256 16424 tablet_bootstrap.cc:492] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: Bootstrap starting.
I20260923 17:29:02.084220 16421 tablet_bootstrap.cc:492] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: Bootstrap starting.
I20260923 17:29:02.084908 16422 tablet_bootstrap.cc:492] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: No bootstrap required, opened a new log
I20260923 17:29:02.085000 16424 tablet_bootstrap.cc:654] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.085031 16422 ts_tablet_manager.cc:1397] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260923 17:29:02.085676 16422 raft_consensus.cc:348] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.085814 16421 tablet_bootstrap.cc:654] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.085855 16422 raft_consensus.cc:374] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.085901 16422 raft_consensus.cc:729] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c0838c331e4d16b93b5991368190fc, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.086105 16422 consensus_queue.cc:260] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.086604 16422 ts_tablet_manager.cc:1428] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:29:02.086503 16439 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.086791 16439 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:29:02.088907 16421 tablet_bootstrap.cc:492] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: No bootstrap required, opened a new log
I20260923 17:29:02.089090 16421 ts_tablet_manager.cc:1397] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:29:02.089466 16424 tablet_bootstrap.cc:492] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: No bootstrap required, opened a new log
I20260923 17:29:02.089617 16424 ts_tablet_manager.cc:1397] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260923 17:29:02.089812 16421 raft_consensus.cc:348] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.089996 16421 raft_consensus.cc:374] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.090088 16421 raft_consensus.cc:729] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749f0bdd7405488998e8a2bcc6c3de95, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.090255 16424 raft_consensus.cc:348] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.090488 16424 raft_consensus.cc:374] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.090569 16424 raft_consensus.cc:729] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bb29f2d86245268821153dc10335b8, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.090564 16421 consensus_queue.cc:260] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.090830 16424 consensus_queue.cc:260] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.091009 16427 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.091282 16427 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:29:02.091396 16424 ts_tablet_manager.cc:1428] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 17:29:02.092015 16429 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.092319 16429 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:29:02.093168 16421 ts_tablet_manager.cc:1428] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260923 17:29:02.096330 16427 raft_consensus.cc:482] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:29:02.096487 16427 raft_consensus.cc:504] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.097839 16427 leader_election.cc:288] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605), 93c0838c331e4d16b93b5991368190fc (127.8.29.195:33765)
I20260923 17:29:02.105203 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.098017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42605 (local address 127.8.29.194:41585)
0923 17:29:02.098929 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.098948 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.099095 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.099485 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.099491 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.099530 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:02.099858 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.099870 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.100900 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.100906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.103128 (+ 2222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.103141 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.104479 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.104484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.104663 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.104699 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.104774 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.104913 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":643,"threads_started":1}
I20260923 17:29:02.105492 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.098163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:41585 (local address 127.8.29.193:42605)
0923 17:29:02.098721 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.098727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.099207 (+ 480us) server_negotiation.cc:407] Connection header received
0923 17:29:02.099277 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.099282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.099349 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.099434 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:02.100018 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.100785 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.103370 (+ 2585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.104326 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.104499 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.104574 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.104827 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.104991 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":419,"thread_start_us":282,"threads_started":1}
I20260923 17:29:02.106127 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca0c6b3346f4eaaae07966629353c3b" candidate_uuid: "749f0bdd7405488998e8a2bcc6c3de95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bb29f2d86245268821153dc10335b8" is_pre_election: true
I20260923 17:29:02.106369 16069 raft_consensus.cc:2454] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 749f0bdd7405488998e8a2bcc6c3de95 in term 0.
I20260923 17:29:02.106856 16139 leader_election.cc:302] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 749f0bdd7405488998e8a2bcc6c3de95, a5bb29f2d86245268821153dc10335b8; no voters:
I20260923 17:29:02.107254 16427 raft_consensus.cc:2781] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:29:02.107379 16427 raft_consensus.cc:482] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:29:02.107470 16427 raft_consensus.cc:3037] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.109095 16462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.098117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:33765 (local address 127.8.29.194:39291)
0923 17:29:02.099204 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.099219 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.099306 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.103128 (+ 3822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.103133 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.103167 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:02.103549 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.103562 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.105271 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.105276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.106939 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.106952 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.108467 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.108472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.108614 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.108639 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.108723 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.108821 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":824,"threads_started":1}
I20260923 17:29:02.109179 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.098522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:39291 (local address 127.8.29.195:33765)
0923 17:29:02.102864 (+ 4342us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.102869 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.102886 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:29:02.102945 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.102948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.103006 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.103097 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:02.104060 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.105145 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.107090 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.108258 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.108415 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.108487 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.108759 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.108891 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4219,"thread_start_us":4101,"threads_started":1}
I20260923 17:29:02.110318 16427 raft_consensus.cc:504] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.111301 16427 leader_election.cc:288] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [CANDIDATE]: Term 1 election: Requested vote from peers a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605), 93c0838c331e4d16b93b5991368190fc (127.8.29.195:33765)
I20260923 17:29:02.111256 16352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca0c6b3346f4eaaae07966629353c3b" candidate_uuid: "749f0bdd7405488998e8a2bcc6c3de95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c0838c331e4d16b93b5991368190fc" is_pre_election: true
I20260923 17:29:02.111425 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca0c6b3346f4eaaae07966629353c3b" candidate_uuid: "749f0bdd7405488998e8a2bcc6c3de95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bb29f2d86245268821153dc10335b8"
I20260923 17:29:02.111650 16069 raft_consensus.cc:3037] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.111712 16352 raft_consensus.cc:2454] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 749f0bdd7405488998e8a2bcc6c3de95 in term 0.
I20260923 17:29:02.113770 16069 raft_consensus.cc:2454] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 749f0bdd7405488998e8a2bcc6c3de95 in term 1.
I20260923 17:29:02.114174 16139 leader_election.cc:302] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 749f0bdd7405488998e8a2bcc6c3de95, a5bb29f2d86245268821153dc10335b8; no voters:
I20260923 17:29:02.114137 16351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eca0c6b3346f4eaaae07966629353c3b" candidate_uuid: "749f0bdd7405488998e8a2bcc6c3de95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c0838c331e4d16b93b5991368190fc"
I20260923 17:29:02.114351 16351 raft_consensus.cc:3037] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.114475 16427 raft_consensus.cc:2781] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:02.114943 16427 raft_consensus.cc:686] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 LEADER]: Becoming Leader. State: Replica: 749f0bdd7405488998e8a2bcc6c3de95, State: Running, Role: LEADER
I20260923 17:29:02.115617 16427 consensus_queue.cc:237] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.116771 16351 raft_consensus.cc:2454] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 749f0bdd7405488998e8a2bcc6c3de95 in term 1.
I20260923 17:29:02.117659 16465 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: TxnStatusTablet state changed. Reason: New leader 749f0bdd7405488998e8a2bcc6c3de95. Latest consensus state: current_term: 1 leader_uuid: "749f0bdd7405488998e8a2bcc6c3de95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.118043 16465 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:29:02.118757 16467 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 17:29:02.118978 16467 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 17:29:02.119602 15912 catalog_manager.cc:5495] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194). New cstate: current_term: 1 leader_uuid: "749f0bdd7405488998e8a2bcc6c3de95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } health_report { overall_health: UNKNOWN } } }
I20260923 17:29:02.141665 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.135736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34677 (local address 127.0.0.1:42900)
0923 17:29:02.135972 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.135991 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.136139 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.136750 (+ 611us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.136755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.136768 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:29:02.136987 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.136999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.138102 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.138106 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.139176 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.139189 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.139914 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.139920 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.140082 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.140145 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.141058 (+ 913us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.141104 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.141200 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.141364 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 17:29:02.141685 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.135772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42900 (local address 127.8.29.194:34677)
0923 17:29:02.136387 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.136392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.136406 (+ 14us) server_negotiation.cc:407] Connection header received
0923 17:29:02.136487 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.136491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.136571 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.136706 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:02.137165 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.137959 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.139320 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.139762 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.139873 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.140371 (+ 498us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.140947 (+ 576us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.141087 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.141254 (+ 167us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.141340 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":308,"threads_started":1}
I20260923 17:29:02.146363 16069 raft_consensus.cc:1261] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Refusing update from remote peer 749f0bdd7405488998e8a2bcc6c3de95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:02.146762 16351 raft_consensus.cc:1261] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Refusing update from remote peer 749f0bdd7405488998e8a2bcc6c3de95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:02.147184 16465 consensus_queue.cc:1035] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:02.147481 16427 consensus_queue.cc:1035] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:02.153831 16429 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: TxnStatusTablet state changed. Reason: New leader 749f0bdd7405488998e8a2bcc6c3de95. Latest consensus state: current_term: 1 leader_uuid: "749f0bdd7405488998e8a2bcc6c3de95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.154192 16429 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:29:02.155925 16439 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: TxnStatusTablet state changed. Reason: New leader 749f0bdd7405488998e8a2bcc6c3de95. Latest consensus state: current_term: 1 leader_uuid: "749f0bdd7405488998e8a2bcc6c3de95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.156236 16439 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:29:02.157466 16472 mvcc.cc:205] Tried to move back new op lower bound from 7332595884623867904 to 7332595884509700096. Current Snapshot: MvccSnapshot[applied={T|T < 7332595884623867904}]
I20260923 17:29:02.163079 16429 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "749f0bdd7405488998e8a2bcc6c3de95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.163363 16429 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P a5bb29f2d86245268821153dc10335b8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:29:02.164872 16427 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "749f0bdd7405488998e8a2bcc6c3de95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.165187 16427 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:29:02.167735 16465 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "749f0bdd7405488998e8a2bcc6c3de95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.168040 16465 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P 749f0bdd7405488998e8a2bcc6c3de95: This TxnStatusTablet replica's current role is: LEADER
I20260923 17:29:02.172860 16439 tablet_replica.cc:439] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "749f0bdd7405488998e8a2bcc6c3de95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } } }
I20260923 17:29:02.173146 16439 tablet_replica.cc:442] T eca0c6b3346f4eaaae07966629353c3b P 93c0838c331e4d16b93b5991368190fc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 17:29:02.186267 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.181096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:50439 (local address 127.8.29.194:34677)
0923 17:29:02.181256 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.181263 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.181667 (+ 404us) server_negotiation.cc:407] Connection header received
0923 17:29:02.181844 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.181848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.181915 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.181987 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:02.182442 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.183236 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.184521 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.184970 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.185077 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.185562 (+ 485us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.185666 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.185739 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.185885 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.185965 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:29:02.186336 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.181032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34677 (local address 127.8.29.195:50439)
0923 17:29:02.181565 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.181581 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.181720 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.182032 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.182038 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.182050 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:29:02.182300 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.182308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.183378 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.183382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.184376 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.184387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.185127 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.185133 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.185272 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.185337 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.185760 (+ 423us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.185771 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.185842 (+ 71us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.185980 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":285,"threads_started":1}
I20260923 17:29:02.196477 16069 raft_consensus.cc:1261] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Refusing update from remote peer 93c0838c331e4d16b93b5991368190fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:02.196509 16208 raft_consensus.cc:1261] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Refusing update from remote peer 93c0838c331e4d16b93b5991368190fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:02.197026 16428 consensus_queue.cc:1035] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:02.197289 16439 consensus_queue.cc:1035] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:02.241791 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.228983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:33765 (local address 127.8.29.194:46649)
0923 17:29:02.235051 (+ 6068us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.235072 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.235439 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.235867 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.235870 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.235881 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:29:02.236218 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.236227 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.237786 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.237792 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.239315 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.239330 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.240566 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.240572 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.240696 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.240757 (+ 61us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.241262 (+ 505us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.241273 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.241349 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.241465 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5832,"thread_start_us":6049,"threads_started":1}
I20260923 17:29:02.241784 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.229069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:46649 (local address 127.8.29.195:33765)
0923 17:29:02.229282 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.229289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.235158 (+ 5869us) server_negotiation.cc:407] Connection header received
0923 17:29:02.235615 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.235621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.235718 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.235822 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:02.236371 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.237468 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.239623 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.240302 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.240442 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.240992 (+ 550us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.241148 (+ 156us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.241263 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.241392 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.241485 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 17:29:02.279037 15912 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:02.279356 15912 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51544:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:02.282814 15912 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252: Sending DeleteTablet for 3 replicas of tablet d1c7861b90bc4d898a924f8769f2c2f4
I20260923 17:29:02.283730 16049 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c7861b90bc4d898a924f8769f2c2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:02 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:52480
I20260923 17:29:02.283842 16188 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c7861b90bc4d898a924f8769f2c2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:02 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:42888
I20260923 17:29:02.284108 16327 tablet_service.cc:1514] Processing DeleteTablet for tablet d1c7861b90bc4d898a924f8769f2c2f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:02 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:38942
I20260923 17:29:02.306661 16491 tablet_replica.cc:330] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: stopping tablet replica
I20260923 17:29:02.306955 16491 raft_consensus.cc:2229] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:29:02.307448 16491 raft_consensus.cc:2258] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:02.308648 16491 ts_tablet_manager.cc:1905] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:02.315026 16490 tablet_replica.cc:330] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: stopping tablet replica
I20260923 17:29:02.315359 16490 raft_consensus.cc:2229] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:02.315713 16490 raft_consensus.cc:2258] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:02.316681 16489 tablet_replica.cc:330] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: stopping tablet replica
I20260923 17:29:02.316884 16489 raft_consensus.cc:2229] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:02.316931 16490 ts_tablet_manager.cc:1905] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:02.317154 16489 raft_consensus.cc:2258] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:02.320322 16489 ts_tablet_manager.cc:1905] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:02.324872 16491 ts_tablet_manager.cc:1918] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:29:02.326664 16491 log.cc:1196] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/d1c7861b90bc4d898a924f8769f2c2f4
I20260923 17:29:02.329097 16490 ts_tablet_manager.cc:1918] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:29:02.329246 16490 log.cc:1196] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/d1c7861b90bc4d898a924f8769f2c2f4
I20260923 17:29:02.329685 16490 ts_tablet_manager.cc:1939] T d1c7861b90bc4d898a924f8769f2c2f4 P 749f0bdd7405488998e8a2bcc6c3de95: Deleting consensus metadata
I20260923 17:29:02.331081 15896 catalog_manager.cc:4841] TS 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677): tablet d1c7861b90bc4d898a924f8769f2c2f4 (table test-table [id=fa4ff00212d64cae98d5ebbfa3e1cf8a]) successfully deleted
I20260923 17:29:02.331223 16491 ts_tablet_manager.cc:1939] T d1c7861b90bc4d898a924f8769f2c2f4 P 93c0838c331e4d16b93b5991368190fc: Deleting consensus metadata
I20260923 17:29:02.332403 15897 catalog_manager.cc:4841] TS 93c0838c331e4d16b93b5991368190fc (127.8.29.195:33765): tablet d1c7861b90bc4d898a924f8769f2c2f4 (table test-table [id=fa4ff00212d64cae98d5ebbfa3e1cf8a]) successfully deleted
I20260923 17:29:02.333745 16489 ts_tablet_manager.cc:1918] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:29:02.333878 16489 log.cc:1196] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/d1c7861b90bc4d898a924f8769f2c2f4
I20260923 17:29:02.334347 16489 ts_tablet_manager.cc:1939] T d1c7861b90bc4d898a924f8769f2c2f4 P a5bb29f2d86245268821153dc10335b8: Deleting consensus metadata
I20260923 17:29:02.335479 15896 catalog_manager.cc:4841] TS a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605): tablet d1c7861b90bc4d898a924f8769f2c2f4 (table test-table [id=fa4ff00212d64cae98d5ebbfa3e1cf8a]) successfully deleted
Sep 23 17:29:02 dist-test-slave-p7jt krb5kdc[15867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 17:29:02 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:02 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:29:02.374503 16456 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:02.361590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51564 (local address 127.8.29.254:35673)
0923 17:29:02.361788 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.361795 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.362328 (+ 533us) server_negotiation.cc:407] Connection header received
0923 17:29:02.362507 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.362513 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.362609 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.362714 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:02.364311 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.365422 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.367299 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.368027 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.368160 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.374196 (+ 6036us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51564: BlockingRecv error: recv got EOF from 127.0.0.1:51564 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 17:29:02 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM
I20260923 17:29:02.394909 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.378623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51566 (local address 127.8.29.254:35673)
0923 17:29:02.378803 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.378809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.378830 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:29:02.378896 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.378901 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.378977 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.379081 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:02.380617 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.381749 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.383582 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.384197 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.384319 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.388754 (+ 4435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:02.388762 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:02.388768 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:02.388823 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:02.392113 (+ 3290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.392900 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.392907 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.392912 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.393078 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.393439 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.393445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.393448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.393816 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:02.394074 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.394271 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.394561 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:29:02.398768 15912 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:29:02.399471 15912 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:29:02.406839 16049 tablet_service.cc:1467] Processing CreateTablet for tablet 202d95177e9b41478b39105396bde8e0 (DEFAULT_TABLE table=test-table [id=846e75e2b99642fab7be2b105235d8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:02.406937 16327 tablet_service.cc:1467] Processing CreateTablet for tablet 202d95177e9b41478b39105396bde8e0 (DEFAULT_TABLE table=test-table [id=846e75e2b99642fab7be2b105235d8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:02.407320 16327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202d95177e9b41478b39105396bde8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.407325 16049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202d95177e9b41478b39105396bde8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.411286 16188 tablet_service.cc:1467] Processing CreateTablet for tablet 202d95177e9b41478b39105396bde8e0 (DEFAULT_TABLE table=test-table [id=846e75e2b99642fab7be2b105235d8a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:02.411710 16188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 202d95177e9b41478b39105396bde8e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.413388 16422 tablet_bootstrap.cc:492] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: Bootstrap starting.
I20260923 17:29:02.415076 16422 tablet_bootstrap.cc:654] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.416095 16424 tablet_bootstrap.cc:492] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: Bootstrap starting.
I20260923 17:29:02.417619 16424 tablet_bootstrap.cc:654] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.420343 16424 tablet_bootstrap.cc:492] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: No bootstrap required, opened a new log
I20260923 17:29:02.420499 16424 ts_tablet_manager.cc:1397] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260923 17:29:02.420616 16421 tablet_bootstrap.cc:492] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: Bootstrap starting.
I20260923 17:29:02.421052 16422 tablet_bootstrap.cc:492] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: No bootstrap required, opened a new log
I20260923 17:29:02.421198 16422 ts_tablet_manager.cc:1397] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.003s
I20260923 17:29:02.421371 16424 raft_consensus.cc:348] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.421618 16424 raft_consensus.cc:374] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.421706 16424 raft_consensus.cc:729] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bb29f2d86245268821153dc10335b8, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.421936 16424 consensus_queue.cc:260] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.421994 16422 raft_consensus.cc:348] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.422230 16422 raft_consensus.cc:374] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.422240 16421 tablet_bootstrap.cc:654] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.422317 16422 raft_consensus.cc:729] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c0838c331e4d16b93b5991368190fc, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.422592 16422 consensus_queue.cc:260] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.423267 16422 ts_tablet_manager.cc:1428] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 17:29:02.427627 16424 ts_tablet_manager.cc:1428] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20260923 17:29:02.429841 16485 raft_consensus.cc:482] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:29:02.430068 16485 raft_consensus.cc:504] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.430924 16485 leader_election.cc:288] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605), 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677)
I20260923 17:29:02.432173 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202d95177e9b41478b39105396bde8e0" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bb29f2d86245268821153dc10335b8" is_pre_election: true
I20260923 17:29:02.432180 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202d95177e9b41478b39105396bde8e0" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f0bdd7405488998e8a2bcc6c3de95" is_pre_election: true
I20260923 17:29:02.432425 16069 raft_consensus.cc:2454] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 0.
W20260923 17:29:02.433053 16278 leader_election.cc:341] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677): Illegal state: must be running to vote when last-logged opid is not known
I20260923 17:29:02.433344 16278 leader_election.cc:302] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 93c0838c331e4d16b93b5991368190fc, a5bb29f2d86245268821153dc10335b8; no voters: 749f0bdd7405488998e8a2bcc6c3de95
I20260923 17:29:02.433717 16421 tablet_bootstrap.cc:492] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: No bootstrap required, opened a new log
I20260923 17:29:02.433876 16421 ts_tablet_manager.cc:1397] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260923 17:29:02.434106 16485 raft_consensus.cc:2781] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:29:02.434221 16485 raft_consensus.cc:482] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:29:02.434320 16485 raft_consensus.cc:3037] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.434798 16421 raft_consensus.cc:348] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.435094 16421 raft_consensus.cc:374] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.435191 16421 raft_consensus.cc:729] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749f0bdd7405488998e8a2bcc6c3de95, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.435420 16421 consensus_queue.cc:260] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.435997 16421 ts_tablet_manager.cc:1428] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 17:29:02.437083 16485 raft_consensus.cc:504] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.437640 16485 leader_election.cc:288] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 election: Requested vote from peers a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605), 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677)
I20260923 17:29:02.438619 16069 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202d95177e9b41478b39105396bde8e0" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a5bb29f2d86245268821153dc10335b8"
I20260923 17:29:02.438835 16069 raft_consensus.cc:3037] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.441099 16069 raft_consensus.cc:2454] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 1.
I20260923 17:29:02.441519 16278 leader_election.cc:302] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93c0838c331e4d16b93b5991368190fc, a5bb29f2d86245268821153dc10335b8; no voters:
I20260923 17:29:02.441615 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "202d95177e9b41478b39105396bde8e0" candidate_uuid: "93c0838c331e4d16b93b5991368190fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f0bdd7405488998e8a2bcc6c3de95"
I20260923 17:29:02.441833 16485 raft_consensus.cc:2781] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:02.441855 16208 raft_consensus.cc:3037] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.441999 16485 raft_consensus.cc:686] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 1 LEADER]: Becoming Leader. State: Replica: 93c0838c331e4d16b93b5991368190fc, State: Running, Role: LEADER
I20260923 17:29:02.442283 16485 consensus_queue.cc:237] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.444988 16208 raft_consensus.cc:2454] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 93c0838c331e4d16b93b5991368190fc in term 1.
I20260923 17:29:02.447324 15911 catalog_manager.cc:5495] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 93c0838c331e4d16b93b5991368190fc (127.8.29.195). New cstate: current_term: 1 leader_uuid: "93c0838c331e4d16b93b5991368190fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } health_report { overall_health: HEALTHY } } }
I20260923 17:29:02.466821 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.462230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38964 (local address 127.8.29.195:33765)
0923 17:29:02.462427 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.462433 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.462452 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:29:02.462557 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.462560 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.462623 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.462695 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:02.463063 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.463847 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.465140 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.465566 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.465674 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.466082 (+ 408us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.466186 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.466259 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.466418 (+ 159us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.466534 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:29:02.469803 16069 raft_consensus.cc:1261] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Refusing update from remote peer 93c0838c331e4d16b93b5991368190fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:02.469803 16208 raft_consensus.cc:1261] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Refusing update from remote peer 93c0838c331e4d16b93b5991368190fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:02.470430 16485 consensus_queue.cc:1035] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:02.470785 16428 consensus_queue.cc:1035] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:02.487387 15910 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:02.487660 15910 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:02.490594 15910 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252: Sending DeleteTablet for 3 replicas of tablet 202d95177e9b41478b39105396bde8e0
I20260923 17:29:02.491379 16049 tablet_service.cc:1514] Processing DeleteTablet for tablet 202d95177e9b41478b39105396bde8e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:02 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:52480
I20260923 17:29:02.491447 16188 tablet_service.cc:1514] Processing DeleteTablet for tablet 202d95177e9b41478b39105396bde8e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:02 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:42888
I20260923 17:29:02.491441 16327 tablet_service.cc:1514] Processing DeleteTablet for tablet 202d95177e9b41478b39105396bde8e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:02 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:38942
I20260923 17:29:02.491667 16489 tablet_replica.cc:330] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: stopping tablet replica
I20260923 17:29:02.491700 16490 tablet_replica.cc:330] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: stopping tablet replica
I20260923 17:29:02.491701 16491 tablet_replica.cc:330] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: stopping tablet replica
I20260923 17:29:02.491860 16489 raft_consensus.cc:2229] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:02.491873 16490 raft_consensus.cc:2229] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:02.491882 16491 raft_consensus.cc:2229] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:29:02.492060 16490 raft_consensus.cc:2258] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:02.492071 16489 raft_consensus.cc:2258] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:02.492128 16491 raft_consensus.cc:2258] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:02.493180 16489 ts_tablet_manager.cc:1905] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:02.493180 16491 ts_tablet_manager.cc:1905] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:02.496830 16490 ts_tablet_manager.cc:1905] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:02.499672 16489 ts_tablet_manager.cc:1918] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:29:02.499691 16491 ts_tablet_manager.cc:1918] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:29:02.499835 16491 log.cc:1196] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/202d95177e9b41478b39105396bde8e0
I20260923 17:29:02.499835 16489 log.cc:1196] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/202d95177e9b41478b39105396bde8e0
I20260923 17:29:02.500209 16489 ts_tablet_manager.cc:1939] T 202d95177e9b41478b39105396bde8e0 P a5bb29f2d86245268821153dc10335b8: Deleting consensus metadata
I20260923 17:29:02.500209 16491 ts_tablet_manager.cc:1939] T 202d95177e9b41478b39105396bde8e0 P 93c0838c331e4d16b93b5991368190fc: Deleting consensus metadata
I20260923 17:29:02.501283 15897 catalog_manager.cc:4841] TS 93c0838c331e4d16b93b5991368190fc (127.8.29.195:33765): tablet 202d95177e9b41478b39105396bde8e0 (table test-table [id=846e75e2b99642fab7be2b105235d8a6]) successfully deleted
I20260923 17:29:02.501343 15896 catalog_manager.cc:4841] TS a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605): tablet 202d95177e9b41478b39105396bde8e0 (table test-table [id=846e75e2b99642fab7be2b105235d8a6]) successfully deleted
I20260923 17:29:02.502741 15910 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51566:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:29:02.503353 15910 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:29:02.504678 16490 ts_tablet_manager.cc:1918] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:29:02.505452 16490 log.cc:1196] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/202d95177e9b41478b39105396bde8e0
I20260923 17:29:02.505918 16490 ts_tablet_manager.cc:1939] T 202d95177e9b41478b39105396bde8e0 P 749f0bdd7405488998e8a2bcc6c3de95: Deleting consensus metadata
I20260923 17:29:02.507040 15896 catalog_manager.cc:4841] TS 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677): tablet 202d95177e9b41478b39105396bde8e0 (table test-table [id=846e75e2b99642fab7be2b105235d8a6]) successfully deleted
I20260923 17:29:02.511248 16049 tablet_service.cc:1467] Processing CreateTablet for tablet c55a54d7ef7b4885a1b9ed1b6d6f946f (DEFAULT_TABLE table=test-table [id=c0215f0acb734c67a8398d79d4292fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:02.511582 16327 tablet_service.cc:1467] Processing CreateTablet for tablet c55a54d7ef7b4885a1b9ed1b6d6f946f (DEFAULT_TABLE table=test-table [id=c0215f0acb734c67a8398d79d4292fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:02.511729 16049 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55a54d7ef7b4885a1b9ed1b6d6f946f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.511731 16188 tablet_service.cc:1467] Processing CreateTablet for tablet c55a54d7ef7b4885a1b9ed1b6d6f946f (DEFAULT_TABLE table=test-table [id=c0215f0acb734c67a8398d79d4292fc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:02.512010 16327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55a54d7ef7b4885a1b9ed1b6d6f946f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.512145 16188 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c55a54d7ef7b4885a1b9ed1b6d6f946f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:02.518684 16424 tablet_bootstrap.cc:492] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: Bootstrap starting.
I20260923 17:29:02.519110 16422 tablet_bootstrap.cc:492] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: Bootstrap starting.
I20260923 17:29:02.520515 16424 tablet_bootstrap.cc:654] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.522492 16422 tablet_bootstrap.cc:654] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.527294 16421 tablet_bootstrap.cc:492] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: Bootstrap starting.
I20260923 17:29:02.529354 16421 tablet_bootstrap.cc:654] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:02.535657 16422 tablet_bootstrap.cc:492] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: No bootstrap required, opened a new log
I20260923 17:29:02.535852 16422 ts_tablet_manager.cc:1397] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.008s
I20260923 17:29:02.536659 16424 tablet_bootstrap.cc:492] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: No bootstrap required, opened a new log
I20260923 17:29:02.536720 16422 raft_consensus.cc:348] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.536971 16422 raft_consensus.cc:374] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.536912 16424 ts_tablet_manager.cc:1397] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.012s
I20260923 17:29:02.537050 16422 raft_consensus.cc:729] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93c0838c331e4d16b93b5991368190fc, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.537310 16422 consensus_queue.cc:260] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.537940 16424 raft_consensus.cc:348] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.538249 16421 tablet_bootstrap.cc:492] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: No bootstrap required, opened a new log
I20260923 17:29:02.538249 16424 raft_consensus.cc:374] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.538416 16424 raft_consensus.cc:729] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bb29f2d86245268821153dc10335b8, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.538431 16421 ts_tablet_manager.cc:1397] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.005s
I20260923 17:29:02.538756 16424 consensus_queue.cc:260] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.539021 16422 ts_tablet_manager.cc:1428] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 17:29:02.539502 16424 ts_tablet_manager.cc:1428] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 17:29:02.540594 16421 raft_consensus.cc:348] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.540807 16421 raft_consensus.cc:374] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:02.542004 16421 raft_consensus.cc:729] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 749f0bdd7405488998e8a2bcc6c3de95, State: Initialized, Role: FOLLOWER
I20260923 17:29:02.542413 16421 consensus_queue.cc:260] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.544730 16421 ts_tablet_manager.cc:1428] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20260923 17:29:02.561810 16429 raft_consensus.cc:482] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:29:02.562039 16429 raft_consensus.cc:504] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.563616 16429 leader_election.cc:288] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677), 93c0838c331e4d16b93b5991368190fc (127.8.29.195:33765)
I20260923 17:29:02.571213 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.563986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:38945 (local address 127.8.29.195:33765)
0923 17:29:02.564172 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.564178 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.565121 (+ 943us) server_negotiation.cc:407] Connection header received
0923 17:29:02.565285 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.565289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.565373 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.565436 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:02.566128 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.566841 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.569098 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.570117 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.570286 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.570373 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.570763 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.570923 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:29:02.572007 16515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.564455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:33765 (local address 127.8.29.193:38945)
0923 17:29:02.565021 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.565033 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.565138 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.565645 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.565650 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.565671 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:02.565985 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.565995 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.566991 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.566995 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.568938 (+ 1943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.568952 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.570338 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.570345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.570555 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.570591 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.571472 (+ 881us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.571627 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":38,"thread_start_us":274,"threads_started":1}
I20260923 17:29:02.572156 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.563990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:48163 (local address 127.8.29.194:34677)
0923 17:29:02.564314 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.564320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.564650 (+ 330us) server_negotiation.cc:407] Connection header received
0923 17:29:02.564995 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.565001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.565085 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.565180 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:02.565835 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.566959 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.569528 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.570860 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.571016 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.571089 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.571659 (+ 570us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.571808 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20260923 17:29:02.572629 16352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55a54d7ef7b4885a1b9ed1b6d6f946f" candidate_uuid: "a5bb29f2d86245268821153dc10335b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c0838c331e4d16b93b5991368190fc" is_pre_election: true
I20260923 17:29:02.572919 16514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.563940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34677 (local address 127.8.29.193:48163)
0923 17:29:02.564551 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.564567 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.564773 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.565255 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.565260 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.565284 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:02.565655 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.565666 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.567102 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.567107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.569363 (+ 2256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.569379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.571274 (+ 1895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.571281 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.571450 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.571483 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.572452 (+ 969us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.572614 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"mutex_wait_us":3,"thread_start_us":323,"threads_started":1}
I20260923 17:29:02.572906 16352 raft_consensus.cc:2454] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5bb29f2d86245268821153dc10335b8 in term 0.
I20260923 17:29:02.573449 16000 leader_election.cc:302] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93c0838c331e4d16b93b5991368190fc, a5bb29f2d86245268821153dc10335b8; no voters:
I20260923 17:29:02.573616 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55a54d7ef7b4885a1b9ed1b6d6f946f" candidate_uuid: "a5bb29f2d86245268821153dc10335b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f0bdd7405488998e8a2bcc6c3de95" is_pre_election: true
I20260923 17:29:02.573904 16208 raft_consensus.cc:2454] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5bb29f2d86245268821153dc10335b8 in term 0.
I20260923 17:29:02.573936 16429 raft_consensus.cc:2781] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:29:02.574039 16429 raft_consensus.cc:482] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:29:02.574148 16429 raft_consensus.cc:3037] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.577201 16429 raft_consensus.cc:504] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.577919 16429 leader_election.cc:288] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [CANDIDATE]: Term 1 election: Requested vote from peers 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677), 93c0838c331e4d16b93b5991368190fc (127.8.29.195:33765)
I20260923 17:29:02.578552 16208 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55a54d7ef7b4885a1b9ed1b6d6f946f" candidate_uuid: "a5bb29f2d86245268821153dc10335b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "749f0bdd7405488998e8a2bcc6c3de95"
I20260923 17:29:02.578545 16352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c55a54d7ef7b4885a1b9ed1b6d6f946f" candidate_uuid: "a5bb29f2d86245268821153dc10335b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93c0838c331e4d16b93b5991368190fc"
I20260923 17:29:02.578758 16352 raft_consensus.cc:3037] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.578816 16208 raft_consensus.cc:3037] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:02.581040 16208 raft_consensus.cc:2454] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5bb29f2d86245268821153dc10335b8 in term 1.
I20260923 17:29:02.581529 15999 leader_election.cc:302] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 749f0bdd7405488998e8a2bcc6c3de95, a5bb29f2d86245268821153dc10335b8; no voters:
I20260923 17:29:02.581678 16352 raft_consensus.cc:2454] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5bb29f2d86245268821153dc10335b8 in term 1.
I20260923 17:29:02.581897 16429 raft_consensus.cc:2781] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:02.582407 16429 raft_consensus.cc:686] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 1 LEADER]: Becoming Leader. State: Replica: a5bb29f2d86245268821153dc10335b8, State: Running, Role: LEADER
I20260923 17:29:02.582835 16429 consensus_queue.cc:237] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } }
I20260923 17:29:02.586839 15910 catalog_manager.cc:5495] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5bb29f2d86245268821153dc10335b8 (127.8.29.193). New cstate: current_term: 1 leader_uuid: "a5bb29f2d86245268821153dc10335b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5bb29f2d86245268821153dc10335b8" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42605 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 } health_report { overall_health: UNKNOWN } } }
I20260923 17:29:02.620507 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.615067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52482 (local address 127.8.29.193:42605)
0923 17:29:02.615540 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.615544 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.615559 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:29:02.615638 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.615642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.615724 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.615859 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:02.616184 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.616965 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.618206 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.618726 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.618841 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.619246 (+ 405us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.619862 (+ 616us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.620027 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.620072 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.620146 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":276,"threads_started":1}
I20260923 17:29:02.635128 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.630171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:39805 (local address 127.8.29.194:34677)
0923 17:29:02.630377 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.630384 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.630830 (+ 446us) server_negotiation.cc:407] Connection header received
0923 17:29:02.630993 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.630997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.631066 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.631141 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:02.631580 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.632262 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.633662 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.634057 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.634191 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.634565 (+ 374us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.634699 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.634765 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.634854 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.634921 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:29:02.635218 16521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.630095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34677 (local address 127.8.29.193:39805)
0923 17:29:02.630717 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.630736 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.630857 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.631178 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.631181 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.631192 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:29:02.631425 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.631433 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.632388 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.632392 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.633512 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.633526 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.634215 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.634219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.634341 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.634400 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.634771 (+ 371us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.634779 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.634833 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.634943 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":299,"threads_started":1}
I20260923 17:29:02.644327 16352 raft_consensus.cc:1261] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Refusing update from remote peer a5bb29f2d86245268821153dc10335b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:02.644327 16208 raft_consensus.cc:1261] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Refusing update from remote peer a5bb29f2d86245268821153dc10335b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:02.645058 16516 consensus_queue.cc:1035] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "749f0bdd7405488998e8a2bcc6c3de95" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:02.645404 16429 consensus_queue.cc:1035] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "93c0838c331e4d16b93b5991368190fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 33765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:02.671506 16487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.666131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42605 (local address 127.8.29.194:44947)
0923 17:29:02.666305 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:02.666322 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:02.666404 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:02.666679 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:02.666682 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:02.666691 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 17:29:02.666844 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.666851 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.668077 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.668082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.669535 (+ 1453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:02.669550 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.670330 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.670334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:02.670493 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.670551 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.671017 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.671030 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 17:29:02.671105 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:29:02.671263 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 17:29:02.671468 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.666247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:44947 (local address 127.8.29.193:42605)
0923 17:29:02.666393 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.666398 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.666413 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:29:02.666513 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.666518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.666594 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.666661 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:02.666978 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.667899 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.669674 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.670164 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.670262 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.670725 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:02.670839 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:02.670912 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.671128 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.671213 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:29:02.707638 15911 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:02.707926 15911 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51566:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:02.711230 15911 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2c5b0da48bc242c6bd3dbce8fa8d2252: Sending DeleteTablet for 3 replicas of tablet c55a54d7ef7b4885a1b9ed1b6d6f946f
I20260923 17:29:02.712253 16327 tablet_service.cc:1514] Processing DeleteTablet for tablet c55a54d7ef7b4885a1b9ed1b6d6f946f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:02 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:38942
I20260923 17:29:02.712234 16188 tablet_service.cc:1514] Processing DeleteTablet for tablet c55a54d7ef7b4885a1b9ed1b6d6f946f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:02 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:42888
I20260923 17:29:02.712291 16049 tablet_service.cc:1514] Processing DeleteTablet for tablet c55a54d7ef7b4885a1b9ed1b6d6f946f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:02 UTC) from {username='oryx', principal='oryx/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:52480
I20260923 17:29:02.712541 16490 tablet_replica.cc:330] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: stopping tablet replica
I20260923 17:29:02.712539 16491 tablet_replica.cc:330] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: stopping tablet replica
I20260923 17:29:02.712610 16489 tablet_replica.cc:330] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: stopping tablet replica
I20260923 17:29:02.712724 16490 raft_consensus.cc:2229] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:02.712724 16491 raft_consensus.cc:2229] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:02.712809 16489 raft_consensus.cc:2229] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:29:02.712924 16490 raft_consensus.cc:2258] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:02.712934 16491 raft_consensus.cc:2258] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:02.713346 16489 raft_consensus.cc:2258] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:02.714432 16490 ts_tablet_manager.cc:1905] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:02.714548 16489 ts_tablet_manager.cc:1905] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:02.719250 16490 ts_tablet_manager.cc:1918] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:29:02.719405 16490 log.cc:1196] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/c55a54d7ef7b4885a1b9ed1b6d6f946f
I20260923 17:29:02.719858 16490 ts_tablet_manager.cc:1939] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 749f0bdd7405488998e8a2bcc6c3de95: Deleting consensus metadata
I20260923 17:29:02.720304 16491 ts_tablet_manager.cc:1905] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:02.721487 15896 catalog_manager.cc:4841] TS 749f0bdd7405488998e8a2bcc6c3de95 (127.8.29.194:34677): tablet c55a54d7ef7b4885a1b9ed1b6d6f946f (table test-table [id=c0215f0acb734c67a8398d79d4292fc0]) successfully deleted
I20260923 17:29:02.722425 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 15983
I20260923 17:29:02.725610 16489 ts_tablet_manager.cc:1918] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:29:02.725894 16489 log.cc:1196] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/c55a54d7ef7b4885a1b9ed1b6d6f946f
I20260923 17:29:02.726419 16489 ts_tablet_manager.cc:1939] T c55a54d7ef7b4885a1b9ed1b6d6f946f P a5bb29f2d86245268821153dc10335b8: Deleting consensus metadata
I20260923 17:29:02.727195 16491 ts_tablet_manager.cc:1918] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 17:29:02.727339 16491 log.cc:1196] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/c55a54d7ef7b4885a1b9ed1b6d6f946f
I20260923 17:29:02.727762 16491 ts_tablet_manager.cc:1939] T c55a54d7ef7b4885a1b9ed1b6d6f946f P 93c0838c331e4d16b93b5991368190fc: Deleting consensus metadata
I20260923 17:29:02.727891 15896 catalog_manager.cc:4841] TS a5bb29f2d86245268821153dc10335b8 (127.8.29.193:42605): tablet c55a54d7ef7b4885a1b9ed1b6d6f946f (table test-table [id=c0215f0acb734c67a8398d79d4292fc0]) successfully deleted
I20260923 17:29:02.728802 15897 catalog_manager.cc:4841] TS 93c0838c331e4d16b93b5991368190fc (127.8.29.195:33765): tablet c55a54d7ef7b4885a1b9ed1b6d6f946f (table test-table [id=c0215f0acb734c67a8398d79d4292fc0]) successfully deleted
Sep 23 17:29:02 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.193@KRBTEST.COM
I20260923 17:29:02.748556 16518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.723697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52494 (local address 127.8.29.193:42605)
0923 17:29:02.723923 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.723930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.728738 (+ 4808us) server_negotiation.cc:407] Connection header received
0923 17:29:02.729096 (+ 358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.729101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.729168 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.729265 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:02.730434 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.731258 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.732491 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.732928 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.733035 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.742226 (+ 9191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:02.742242 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:02.742264 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:02.742307 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:02.745512 (+ 3205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.746284 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.746292 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.746294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.746447 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.746866 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.746870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.746872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.747245 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:02.747546 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.747947 (+ 401us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.748181 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:29:02.749241 16114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:02.860339 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 15983
I20260923 17:29:02.877458 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 16123
Sep 23 17:29:02 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.194@KRBTEST.COM
I20260923 17:29:02.893054 16470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:02.878294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42904 (local address 127.8.29.194:34677)
0923 17:29:02.878489 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:02.878496 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:02.878517 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:29:02.878578 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:02.878581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:02.878663 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:02.878746 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:02.880040 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.880773 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.882109 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:02.882605 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:02.882699 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:02.886985 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:02.887036 (+ 51us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:02.887053 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:02.887102 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:02.890033 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.890917 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.890928 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.890932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.891084 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:02.891462 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:02.891467 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:02.891471 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:02.891857 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:02.892150 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:02.892482 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:29:02.892725 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:29:02.893697 16253 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:03.015022 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 16123
I20260923 17:29:03.032794 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 16262
Sep 23 17:29:03 dist-test-slave-p7jt krb5kdc[15867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.29.195@KRBTEST.COM
I20260923 17:29:03.048216 16464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:03.033624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38978 (local address 127.8.29.195:33765)
0923 17:29:03.033817 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:03.033823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:03.033843 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:29:03.034045 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:03.034049 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:03.034154 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:03.034362 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:03.035726 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:03.036399 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:03.037701 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:03.038112 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:03.038226 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:03.042548 (+ 4322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:03.042565 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:03.042582 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:03.042622 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:03.045299 (+ 2677us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:03.046013 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:03.046022 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:03.046024 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:03.046174 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:03.046684 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:03.046688 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:03.046690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:03.047075 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:03.047378 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:03.047682 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 17:29:03.047902 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:29:03.048862 16392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:03.170789 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 16262
I20260923 17:29:03.188526 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 15880
I20260923 17:29:03.189715 15952 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:03.304360 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 15880
2026-09-23T17:29:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (5295 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:03 dist-test-slave-p7jt krb5kdc[16534](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:03 dist-test-slave-p7jt krb5kdc[16534](info): set up 2 sockets
Sep 23 17:29:03 dist-test-slave-p7jt krb5kdc[16534](info): commencing operation
krb5kdc: starting...
W20260923 17:29:04.323223 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.002s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:29:04 dist-test-slave-p7jt krb5kdc[16534](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:29:04.543126 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40763
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:40763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:04.867031 16547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:04.867341 16547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:04.867430 16547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:04.874747 16547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:04.874863 16547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:04.874917 16547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:04.874971 16547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:04.875016 16547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:04.885607 16547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:40763
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:04.887023 16547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:04.888321 16547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:04.900242 16553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:04.900280 16555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:04.900537 16552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:04.901209 16547 server_base.cc:1034] running on GCE node
I20260923 17:29:04.902073 16547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:04.903616 16547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:04.904857 16547 hybrid_clock.cc:648] HybridClock initialized: now 1790184544904798 us; error 36 us; skew 500 ppm
I20260923 17:29:04.905236 16547 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T17:29:04Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1669 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:05 dist-test-slave-p7jt krb5kdc[16561](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:05 dist-test-slave-p7jt krb5kdc[16561](info): set up 2 sockets
Sep 23 17:29:05 dist-test-slave-p7jt krb5kdc[16561](info): commencing operation
krb5kdc: starting...
W20260923 17:29:06.007332 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:29:06 dist-test-slave-p7jt krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:29:06.230815 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44205
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:44205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:06.561774 16574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:06.562080 16574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:06.562199 16574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:06.569607 16574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:06.569731 16574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:06.569787 16574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:06.569865 16574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:06.569921 16574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:06.580173 16574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:44205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:06.581568 16574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:06.582962 16574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:06.595112 16579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:06.595059 16582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:06.595085 16580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:06.596385 16574 server_base.cc:1034] running on GCE node
I20260923 17:29:06.597249 16574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:06.598884 16574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:06.600107 16574 hybrid_clock.cc:648] HybridClock initialized: now 1790184546600054 us; error 46 us; skew 500 ppm
I20260923 17:29:06.600461 16574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:06 dist-test-slave-p7jt krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:06.603600 16574 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:29:06.605060 16574 webserver.cc:460] Webserver started at http://127.8.29.254:39469/ using document root <none> and password file <none>
I20260923 17:29:06.605784 16574 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:06.605890 16574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:06.606204 16574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:06.609241 16574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "05db492582034326bac2a58248e586f8"
format_stamp: "Formatted at 2026-09-23 17:29:06 on dist-test-slave-p7jt"
server_key: "96ef6f6e99c22611271d0280d3dc93a3"
server_key_iv: "2efac988128010c30fa04bd1b04be391"
server_key_version: "encryptionkey@0"
I20260923 17:29:06.610052 16574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "05db492582034326bac2a58248e586f8"
format_stamp: "Formatted at 2026-09-23 17:29:06 on dist-test-slave-p7jt"
server_key: "96ef6f6e99c22611271d0280d3dc93a3"
server_key_iv: "2efac988128010c30fa04bd1b04be391"
server_key_version: "encryptionkey@0"
I20260923 17:29:06.615687 16574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 17:29:06.619370 16589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:06.620865 16574 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:29:06.621022 16574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "05db492582034326bac2a58248e586f8"
format_stamp: "Formatted at 2026-09-23 17:29:06 on dist-test-slave-p7jt"
server_key: "96ef6f6e99c22611271d0280d3dc93a3"
server_key_iv: "2efac988128010c30fa04bd1b04be391"
server_key_version: "encryptionkey@0"
I20260923 17:29:06.621300 16574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:06.647902 16574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:06.652438 16574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:06.652727 16574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:06.670717 16574 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:44205
I20260923 17:29:06.670737 16640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:44205 every 8 connection(s)
I20260923 17:29:06.672127 16574 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:06.672412 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 16574
I20260923 17:29:06.672613 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:29:06.673322 8311 external_mini_cluster.cc:1428] Setting key bcc54544b3e80c3b0d3728aaf9f6b989
I20260923 17:29:06.676360 16641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:29:06 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:29:06.689608 16641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: Bootstrap starting.
I20260923 17:29:06.692948 16641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:06.694126 16641 log.cc:824] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:06.696560 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:06.675309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.8.29.254:44205)
0923 17:29:06.676072 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:06.676085 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:06.676124 (+ 39us) server_negotiation.cc:407] Connection header received
0923 17:29:06.677769 (+ 1645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:06.677792 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:06.678186 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:06.678582 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:06.680364 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:06.681764 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:06.683550 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:06.684287 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:06.684450 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:06.689262 (+ 4812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:06.689271 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:06.689289 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:06.689333 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:06.692264 (+ 2931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:06.693135 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:06.693142 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:06.693149 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:06.693331 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:06.693709 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:06.693711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:06.693713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:06.694267 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:06.694518 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:06.695083 (+ 565us) server_negotiation.cc:299] Negotiation successful
0923 17:29:06.695548 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":332,"threads_started":1}
I20260923 17:29:06.697120 16641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: No bootstrap required, opened a new log
I20260923 17:29:06.705204 16641 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } }
I20260923 17:29:06.705614 16641 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:06.705713 16641 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05db492582034326bac2a58248e586f8, State: Initialized, Role: FOLLOWER
I20260923 17:29:06.706302 16641 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } }
I20260923 17:29:06.706631 16641 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:06.706769 16641 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:06.706954 16641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:06.709067 16641 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } }
I20260923 17:29:06.709555 16641 leader_election.cc:302] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05db492582034326bac2a58248e586f8; no voters:
I20260923 17:29:06.710137 16641 leader_election.cc:288] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:06.710286 16646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:06.710935 16646 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 1 LEADER]: Becoming Leader. State: Replica: 05db492582034326bac2a58248e586f8, State: Running, Role: LEADER
I20260923 17:29:06.711437 16646 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } }
I20260923 17:29:06.712143 16641 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:06.714573 16648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05db492582034326bac2a58248e586f8. Latest consensus state: current_term: 1 leader_uuid: "05db492582034326bac2a58248e586f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } } }
I20260923 17:29:06.715029 16648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:06.714895 16647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05db492582034326bac2a58248e586f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } } }
I20260923 17:29:06.715193 16647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:06.716866 16653 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:06.721966 16653 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:06.731495 16653 catalog_manager.cc:1347] Generated new cluster ID: 6464c49b7a504f9a99e8017a81c060fb
I20260923 17:29:06.731657 16653 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:06.745302 16653 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:06.746716 16653 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:06.757619 16653 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: Generated new TSK 0
I20260923 17:29:06.758605 16653 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:29:06.822624 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:44205
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:07.164657 16667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:07.164950 16667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:07.165073 16667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:07.172410 16667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:07.172547 16667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:07.172714 16667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:29:07.182685 16667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:07.184072 16667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:07.185401 16667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:07.199168 16673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:07.199208 16675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:07.199229 16672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:07.199856 16667 server_base.cc:1034] running on GCE node
I20260923 17:29:07.200493 16667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:07.201452 16667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:07.202673 16667 hybrid_clock.cc:648] HybridClock initialized: now 1790184547202608 us; error 56 us; skew 500 ppm
I20260923 17:29:07.203001 16667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:07 dist-test-slave-p7jt krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184547, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:07.206110 16667 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:29:07.207674 16667 webserver.cc:460] Webserver started at http://127.8.29.193:36839/ using document root <none> and password file <none>
I20260923 17:29:07.208353 16667 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:07.208468 16667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:07.208758 16667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:07.211706 16667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "e38cd473f4284490b1a74443ae39b3ba"
format_stamp: "Formatted at 2026-09-23 17:29:07 on dist-test-slave-p7jt"
server_key: "dd424c47577390d152f49924227c3d2d"
server_key_iv: "60b716069bd80aecaa2617b8744aae43"
server_key_version: "encryptionkey@0"
I20260923 17:29:07.212491 16667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "e38cd473f4284490b1a74443ae39b3ba"
format_stamp: "Formatted at 2026-09-23 17:29:07 on dist-test-slave-p7jt"
server_key: "dd424c47577390d152f49924227c3d2d"
server_key_iv: "60b716069bd80aecaa2617b8744aae43"
server_key_version: "encryptionkey@0"
I20260923 17:29:07.217913 16667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:29:07.221470 16682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:07.222865 16667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:29:07.223016 16667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "e38cd473f4284490b1a74443ae39b3ba"
format_stamp: "Formatted at 2026-09-23 17:29:07 on dist-test-slave-p7jt"
server_key: "dd424c47577390d152f49924227c3d2d"
server_key_iv: "60b716069bd80aecaa2617b8744aae43"
server_key_version: "encryptionkey@0"
I20260923 17:29:07.223204 16667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:07.242082 16667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:07.246416 16667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:07.246722 16667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:07.247687 16667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:07.249102 16667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:07.249173 16667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:07.249258 16667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:07.249300 16667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:07.279453 16667 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:40951
I20260923 17:29:07.279502 16794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:40951 every 8 connection(s)
I20260923 17:29:07.280715 16667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:29:07.287559 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 16667
I20260923 17:29:07.287765 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:29:07.288381 8311 external_mini_cluster.cc:1428] Setting key f768666d7d59bafb78deb30e08561707
Sep 23 17:29:07 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184547, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:29:07.301616 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:07.283282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:49355 (local address 127.8.29.254:44205)
0923 17:29:07.283697 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:07.283703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:07.284638 (+ 935us) server_negotiation.cc:407] Connection header received
0923 17:29:07.286428 (+ 1790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:07.286434 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:07.286552 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:07.286646 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:07.288256 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:07.289040 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:07.291161 (+ 2121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:07.291567 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:07.291671 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:07.295862 (+ 4191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:07.295867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:07.295871 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:07.295907 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:07.298488 (+ 2581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:07.299231 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:07.299236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:07.299240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:07.299376 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:07.299815 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:07.299819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:07.299821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:07.300106 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:07.300242 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:07.301002 (+ 760us) server_negotiation.cc:299] Negotiation successful
0923 17:29:07.301268 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":215,"threads_started":1}
I20260923 17:29:07.302695 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:07.283754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:49355)
0923 17:29:07.284468 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:07.284507 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:07.286153 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:07.286901 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:07.286911 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:07.287439 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:07.288084 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:07.288104 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:07.289245 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:07.289250 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:07.290965 (+ 1715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:07.290982 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:07.291808 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:07.291816 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:07.292130 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:07.292872 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:07.292907 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:07.295592 (+ 2685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:07.298642 (+ 3050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:07.298650 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:07.298665 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:07.299100 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:07.299490 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:07.299494 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:07.299496 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:07.299702 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:07.300232 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:07.300240 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:07.300618 (+ 378us) client_negotiation.cc:769] Sending connection context
0923 17:29:07.300889 (+ 271us) client_negotiation.cc:240] Negotiation successful
0923 17:29:07.301478 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":275,"threads_started":1}
I20260923 17:29:07.304252 16795 heartbeater.cc:347] Connected to a master server at 127.8.29.254:44205
I20260923 17:29:07.304548 16795 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:07.305269 16795 heartbeater.cc:511] Master 127.8.29.254:44205 requested a full tablet report, sending...
I20260923 17:29:07.307643 16606 ts_manager.cc:194] Registered new tserver with Master: e38cd473f4284490b1a74443ae39b3ba (127.8.29.193:40951)
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:29:07.309777 16606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:49355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:29:07.339125 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:44205
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:07.667832 16801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:07.668138 16801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:07.668255 16801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:07.675911 16801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:07.676052 16801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:07.676216 16801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:29:07.686259 16801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:07.687618 16801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:07.688931 16801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:07.702303 16806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:07.702337 16807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:07.702409 16809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:07.703030 16801 server_base.cc:1034] running on GCE node
I20260923 17:29:07.703709 16801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:07.704623 16801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:07.705828 16801 hybrid_clock.cc:648] HybridClock initialized: now 1790184547705777 us; error 42 us; skew 500 ppm
I20260923 17:29:07.706156 16801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:07 dist-test-slave-p7jt krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184547, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:07.709263 16801 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:29:07.710868 16801 webserver.cc:460] Webserver started at http://127.8.29.194:38593/ using document root <none> and password file <none>
I20260923 17:29:07.711602 16801 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:07.711721 16801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:07.712023 16801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:07.715309 16801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "ca697bba16784b7a85d312f0229b587d"
format_stamp: "Formatted at 2026-09-23 17:29:07 on dist-test-slave-p7jt"
server_key: "d8eeae48e5bb675c5001cf399a537f9c"
server_key_iv: "86addcf3bf40bb82a29ccae78e655337"
server_key_version: "encryptionkey@0"
I20260923 17:29:07.716102 16801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "ca697bba16784b7a85d312f0229b587d"
format_stamp: "Formatted at 2026-09-23 17:29:07 on dist-test-slave-p7jt"
server_key: "d8eeae48e5bb675c5001cf399a537f9c"
server_key_iv: "86addcf3bf40bb82a29ccae78e655337"
server_key_version: "encryptionkey@0"
I20260923 17:29:07.721771 16801 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:29:07.725417 16816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:07.727098 16801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:29:07.727262 16801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "ca697bba16784b7a85d312f0229b587d"
format_stamp: "Formatted at 2026-09-23 17:29:07 on dist-test-slave-p7jt"
server_key: "d8eeae48e5bb675c5001cf399a537f9c"
server_key_iv: "86addcf3bf40bb82a29ccae78e655337"
server_key_version: "encryptionkey@0"
I20260923 17:29:07.727456 16801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:07.741026 16801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:07.749573 16801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:07.749891 16801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:07.750952 16801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:07.752372 16801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:07.752442 16801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:07.752543 16801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:07.752599 16801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:07.783537 16801 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:45299
I20260923 17:29:07.783581 16928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:45299 every 8 connection(s)
I20260923 17:29:07.784847 16801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:29:07.790652 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 16801
I20260923 17:29:07.790834 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:29:07.791270 8311 external_mini_cluster.cc:1428] Setting key f2c48462cf914d767a2be513b07955b6
Sep 23 17:29:07 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184547, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:29:07.807891 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:07.787432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:55255 (local address 127.8.29.254:44205)
0923 17:29:07.787636 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:07.787641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:07.788936 (+ 1295us) server_negotiation.cc:407] Connection header received
0923 17:29:07.791010 (+ 2074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:07.791017 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:07.791112 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:07.791236 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:07.793183 (+ 1947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:07.794157 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:07.796548 (+ 2391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:07.797047 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:07.797160 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:07.801485 (+ 4325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:07.801493 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:07.801498 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:07.801566 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:07.804343 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:07.805333 (+ 990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:07.805338 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:07.805340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:07.805489 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:07.805978 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:07.805982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:07.805984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:07.806297 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:07.806409 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:07.807328 (+ 919us) server_negotiation.cc:299] Negotiation successful
0923 17:29:07.807598 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:29:07.809192 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:07.787919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:55255)
0923 17:29:07.788730 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:07.788776 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:07.790648 (+ 1872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:07.791545 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:07.791558 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:07.792287 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:07.792976 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:07.792997 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:07.794348 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:07.794353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:07.796365 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:07.796381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:07.797276 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:07.797283 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:07.797585 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:07.798422 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:07.798452 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:07.801183 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:07.804520 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:07.804541 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:07.804561 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:07.805154 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:07.805606 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:07.805610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:07.805613 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:07.805866 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:07.806417 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:07.806425 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:07.806897 (+ 472us) client_negotiation.cc:769] Sending connection context
0923 17:29:07.807184 (+ 287us) client_negotiation.cc:240] Negotiation successful
0923 17:29:07.807790 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":314,"threads_started":1}
I20260923 17:29:07.810990 16929 heartbeater.cc:347] Connected to a master server at 127.8.29.254:44205
I20260923 17:29:07.811357 16929 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:07.812173 16929 heartbeater.cc:511] Master 127.8.29.254:44205 requested a full tablet report, sending...
I20260923 17:29:07.813967 16606 ts_manager.cc:194] Registered new tserver with Master: ca697bba16784b7a85d312f0229b587d (127.8.29.194:45299)
I20260923 17:29:07.814904 16606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:55255
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:29:07.844069 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:44205
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:08.172917 16934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:08.173206 16934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:08.173298 16934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:08.180599 16934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:08.180718 16934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:08.180831 16934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:29:08.190680 16934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:08.191957 16934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:08.193397 16934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:08.206241 16939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:08.206341 16942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:08.206341 16940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:08.207221 16934 server_base.cc:1034] running on GCE node
I20260923 17:29:08.207800 16934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:08.208771 16934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:08.209993 16934 hybrid_clock.cc:648] HybridClock initialized: now 1790184548209940 us; error 47 us; skew 500 ppm
I20260923 17:29:08.210345 16934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:08 dist-test-slave-p7jt krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184548, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:08.213523 16934 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:29:08.215137 16934 webserver.cc:460] Webserver started at http://127.8.29.195:37519/ using document root <none> and password file <none>
I20260923 17:29:08.215816 16934 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:08.215927 16934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:08.216219 16934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:08.219246 16934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "845d3f7d9e6143b791c3ebcd5129f2d8"
format_stamp: "Formatted at 2026-09-23 17:29:08 on dist-test-slave-p7jt"
server_key: "eddedf30f12909fa8b2f27e1bf03f250"
server_key_iv: "2c41fac05cbc4ca3778b4186885a38f8"
server_key_version: "encryptionkey@0"
I20260923 17:29:08.220006 16934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "845d3f7d9e6143b791c3ebcd5129f2d8"
format_stamp: "Formatted at 2026-09-23 17:29:08 on dist-test-slave-p7jt"
server_key: "eddedf30f12909fa8b2f27e1bf03f250"
server_key_iv: "2c41fac05cbc4ca3778b4186885a38f8"
server_key_version: "encryptionkey@0"
I20260923 17:29:08.225188 16934 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:29:08.228880 16949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:08.230262 16934 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260923 17:29:08.230397 16934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "845d3f7d9e6143b791c3ebcd5129f2d8"
format_stamp: "Formatted at 2026-09-23 17:29:08 on dist-test-slave-p7jt"
server_key: "eddedf30f12909fa8b2f27e1bf03f250"
server_key_iv: "2c41fac05cbc4ca3778b4186885a38f8"
server_key_version: "encryptionkey@0"
I20260923 17:29:08.230657 16934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:08.257031 16934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:08.261140 16934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:08.261390 16934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:08.262301 16934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:08.263654 16934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:08.263729 16934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:08.263818 16934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:08.263859 16934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:08.293606 16934 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:40371
I20260923 17:29:08.293643 17061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:40371 every 8 connection(s)
I20260923 17:29:08.294850 16934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:29:08.305091 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 16934
I20260923 17:29:08.305294 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:29:08.305876 8311 external_mini_cluster.cc:1428] Setting key c7f4f51adb0323d0a1050dcb9529d87a
Sep 23 17:29:08 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184548, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:29:08.313115 16795 heartbeater.cc:503] Master 127.8.29.254:44205 was elected leader, sending a full tablet report...
I20260923 17:29:08.315176 16797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:08.297243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:47537 (local address 127.8.29.254:44205)
0923 17:29:08.297472 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:08.297479 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:08.298693 (+ 1214us) server_negotiation.cc:407] Connection header received
0923 17:29:08.300515 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:08.300520 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:08.300597 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:08.300687 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:08.302202 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.303009 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.304653 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.305057 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.305152 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:08.309340 (+ 4188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:08.309345 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:08.309348 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:08.309382 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:08.311948 (+ 2566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:08.312725 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:08.312731 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:08.312734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:08.312903 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:08.313363 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:08.313368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:08.313372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:08.313664 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:08.313810 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:08.314594 (+ 784us) server_negotiation.cc:299] Negotiation successful
0923 17:29:08.314823 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:29:08.316223 17064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:08.297798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:47537)
0923 17:29:08.298534 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:08.298584 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:08.300204 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:08.300888 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:08.300896 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:08.301402 (+ 506us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:08.302027 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:08.302048 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.303192 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.303196 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:08.304490 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:08.304507 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.305304 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.305313 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:08.305620 (+ 307us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:08.306356 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:08.306387 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:08.309125 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:08.312098 (+ 2973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:08.312108 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:08.312127 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:08.312591 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:08.313016 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:08.313021 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:08.313025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:08.313239 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:08.313796 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:08.313805 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:08.314179 (+ 374us) client_negotiation.cc:769] Sending connection context
0923 17:29:08.314452 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 17:29:08.315059 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":278,"threads_started":1}
I20260923 17:29:08.317607 17062 heartbeater.cc:347] Connected to a master server at 127.8.29.254:44205
I20260923 17:29:08.317906 17062 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:08.318696 17062 heartbeater.cc:511] Master 127.8.29.254:44205 requested a full tablet report, sending...
I20260923 17:29:08.320421 16606 ts_manager.cc:194] Registered new tserver with Master: 845d3f7d9e6143b791c3ebcd5129f2d8 (127.8.29.195:40371)
I20260923 17:29:08.321477 16606 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:47537
I20260923 17:29:08.321919 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab.
Entry for principal oryx/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab.
WARNING: no policy specified for oryx/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab.
Entry for principal oryx/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab.
WARNING: no policy specified for oryx/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab.
Entry for principal oryx/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab.
I20260923 17:29:08.461812 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 16667
Sep 23 17:29:08 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:29:08.479518 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:08.463064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48186 (local address 127.8.29.193:40951)
0923 17:29:08.463699 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:08.463706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:08.463737 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:29:08.463803 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:08.463811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:08.464060 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:08.464214 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:08.465503 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.466711 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.468057 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.468670 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.468779 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:08.473276 (+ 4497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:08.473297 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:08.473320 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:08.473374 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:08.476628 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:08.477494 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:08.477503 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:08.477514 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:08.477685 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:08.478024 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:08.478028 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:08.478030 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:08.478420 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:08.478739 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:08.479001 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 17:29:08.479217 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":308,"threads_started":1}
I20260923 17:29:08.480786 16790 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:08.568524 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 16667
I20260923 17:29:08.582100 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 16801
Sep 23 17:29:08 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:29:08.598637 17074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:08.582949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.8.29.194:45299)
0923 17:29:08.583506 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:08.583521 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:08.583550 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:29:08.583610 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:08.583617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:08.583833 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:08.583962 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:08.585200 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.586331 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.587750 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.588375 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.588487 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:08.593051 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:08.593069 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:08.593092 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:08.593137 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:08.595991 (+ 2854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:08.596756 (+ 765us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:08.596763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:08.596766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:08.596894 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:08.597218 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:08.597221 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:08.597223 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:08.597604 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:08.597841 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:08.598059 (+ 218us) server_negotiation.cc:299] Negotiation successful
0923 17:29:08.598269 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":283,"threads_started":1}
I20260923 17:29:08.599910 16924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:08.682446 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 16801
I20260923 17:29:08.695395 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 16934
Sep 23 17:29:08 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:29:08.711027 17076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:08.696239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41604 (local address 127.8.29.195:40371)
0923 17:29:08.696828 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:08.696835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:08.696865 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:29:08.696925 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:08.696932 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:08.697145 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:08.697273 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:08.698525 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.699557 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.700812 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:08.701360 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:08.701455 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:08.705660 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:08.705673 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:08.705691 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:08.705733 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:08.708375 (+ 2642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:08.709044 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:08.709052 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:08.709054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:08.709182 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:08.709496 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:08.709500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:08.709503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:08.709906 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:08.710152 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:08.710390 (+ 238us) server_negotiation.cc:299] Negotiation successful
0923 17:29:08.710726 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":283,"threads_started":1}
I20260923 17:29:08.712330 17057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:08.794282 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 16934
I20260923 17:29:08.806699 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 16574
I20260923 17:29:08.807896 16636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:08.911821 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 16574
I20260923 17:29:08.928952 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44205
--webserver_interface=127.8.29.254
--webserver_port=39469
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:44205
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:09.251926 17079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:09.252233 17079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:09.252322 17079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:09.260160 17079 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:09.260283 17079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:09.260327 17079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:09.260382 17079 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:09.260423 17079 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:09.270601 17079 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:44205
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44205
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=39469
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:09.271946 17079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:09.273283 17079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:09.286034 17084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:09.286070 17087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:09.286109 17085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:09.286801 17079 server_base.cc:1034] running on GCE node
I20260923 17:29:09.287652 17079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:09.289201 17079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:09.290445 17079 hybrid_clock.cc:648] HybridClock initialized: now 1790184549290378 us; error 56 us; skew 500 ppm
I20260923 17:29:09.290853 17079 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:09 dist-test-slave-p7jt krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184549, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:09.293991 17079 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:29:09.295553 17079 webserver.cc:460] Webserver started at http://127.8.29.254:39469/ using document root <none> and password file <none>
I20260923 17:29:09.296250 17079 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:09.296373 17079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:09.302194 17079 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260923 17:29:09.305224 17094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:09.306779 17079 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:29:09.306949 17079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "05db492582034326bac2a58248e586f8"
format_stamp: "Formatted at 2026-09-23 17:29:06 on dist-test-slave-p7jt"
server_key: "96ef6f6e99c22611271d0280d3dc93a3"
server_key_iv: "2efac988128010c30fa04bd1b04be391"
server_key_version: "encryptionkey@0"
I20260923 17:29:09.307943 17079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:09.329160 17079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:09.333385 17079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:09.333645 17079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:09.350384 17079 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:44205
I20260923 17:29:09.350405 17145 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:44205 every 8 connection(s)
I20260923 17:29:09.352057 17079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:09.359297 17146 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:29:09.360283 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 17079
I20260923 17:29:09.360996 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:40951
--local_ip_for_outbound_sockets=127.8.29.193
--tserver_master_addrs=127.8.29.254:44205
--webserver_port=36839
--webserver_interface=127.8.29.193
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.29.193
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:29:09.362736 17146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: Bootstrap starting.
I20260923 17:29:09.381304 17146 log.cc:824] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:09.390300 17146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:29:09.390864 17146 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: Bootstrap complete.
I20260923 17:29:09.400496 17146 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } }
I20260923 17:29:09.401453 17146 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05db492582034326bac2a58248e586f8, State: Initialized, Role: FOLLOWER
I20260923 17:29:09.402083 17146 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } }
I20260923 17:29:09.402302 17146 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:09.402412 17146 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:09.402592 17146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:29:09.406522 17146 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } }
I20260923 17:29:09.407045 17146 leader_election.cc:302] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05db492582034326bac2a58248e586f8; no voters:
I20260923 17:29:09.407644 17146 leader_election.cc:288] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:29:09.407795 17150 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:29:09.408850 17150 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [term 2 LEADER]: Becoming Leader. State: Replica: 05db492582034326bac2a58248e586f8, State: Running, Role: LEADER
I20260923 17:29:09.409413 17150 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } }
I20260923 17:29:09.409874 17146 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:09.411911 17152 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05db492582034326bac2a58248e586f8. Latest consensus state: current_term: 2 leader_uuid: "05db492582034326bac2a58248e586f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } } }
I20260923 17:29:09.412294 17152 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:09.412989 17151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "05db492582034326bac2a58248e586f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05db492582034326bac2a58248e586f8" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44205 } } }
I20260923 17:29:09.413308 17151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:09.414211 17159 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:09.430327 17159 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:09.431895 17159 catalog_manager.cc:1259] Loaded cluster ID: 6464c49b7a504f9a99e8017a81c060fb
I20260923 17:29:09.432026 17159 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:09.436414 17159 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:09.438086 17159 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 05db492582034326bac2a58248e586f8: Loaded TSK: 0
I20260923 17:29:09.439793 17159 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:29:09.715579 17148 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:09.715883 17148 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:09.716013 17148 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:09.723480 17148 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:09.723620 17148 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:09.723768 17148 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:29:09.733779 17148 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.193.keytab
--principal=oryx/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:40951
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=36839
--tserver_master_addrs=127.8.29.254:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:09.735167 17148 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:09.736500 17148 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:09.751598 17172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:09.751598 17173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:09.752720 17175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:09.752738 17148 server_base.cc:1034] running on GCE node
I20260923 17:29:09.753500 17148 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:09.754612 17148 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:09.755848 17148 hybrid_clock.cc:648] HybridClock initialized: now 1790184549755792 us; error 47 us; skew 500 ppm
I20260923 17:29:09.756222 17148 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:09 dist-test-slave-p7jt krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184549, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:09.759536 17148 init.cc:377] Logged in from keytab as oryx/127.8.29.193@KRBTEST.COM (short username oryx)
I20260923 17:29:09.761332 17148 webserver.cc:460] Webserver started at http://127.8.29.193:36839/ using document root <none> and password file <none>
I20260923 17:29:09.762070 17148 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:09.762216 17148 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:09.768545 17148 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260923 17:29:09.771596 17182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:09.773305 17148 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:29:09.773504 17148 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "e38cd473f4284490b1a74443ae39b3ba"
format_stamp: "Formatted at 2026-09-23 17:29:07 on dist-test-slave-p7jt"
server_key: "dd424c47577390d152f49924227c3d2d"
server_key_iv: "60b716069bd80aecaa2617b8744aae43"
server_key_version: "encryptionkey@0"
I20260923 17:29:09.774618 17148 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:09.789845 17148 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:09.794358 17148 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:09.794673 17148 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:09.795689 17148 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:09.797032 17148 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:09.797130 17148 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:09.797231 17148 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:09.797298 17148 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:09.828233 17148 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:40951
I20260923 17:29:09.828290 17294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:40951 every 8 connection(s)
I20260923 17:29:09.830072 17148 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:29:09.832250 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 17148
I20260923 17:29:09.832837 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:45299
--local_ip_for_outbound_sockets=127.8.29.194
--tserver_master_addrs=127.8.29.254:44205
--webserver_port=38593
--webserver_interface=127.8.29.194
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.29.194
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:29:09 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:09 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:09.851548 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:09.833402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:54477)
0923 17:29:09.835053 (+ 1651us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:09.835107 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:09.837269 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:09.838940 (+ 1671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:09.838950 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:09.839668 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:09.840384 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:09.840403 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.842964 (+ 2561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.842970 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:09.844309 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:09.844327 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.846001 (+ 1674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.846007 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:09.846228 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:09.846931 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:09.846963 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:09.850564 (+ 3601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1354,"thread_start_us":1153,"threads_started":1}
W20260923 17:29:09.852243 17295 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:44205 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.29.254:44205: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
W20260923 17:29:09.852304 17297 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:09.833297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:54477 (local address 127.8.29.254:44205)
0923 17:29:09.834193 (+ 896us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:09.834214 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:09.835263 (+ 1049us) server_negotiation.cc:407] Connection header received
0923 17:29:09.837921 (+ 2658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:09.837942 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:09.838332 (+ 390us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:09.838727 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:09.840633 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.842723 (+ 2090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.844498 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.845723 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.845957 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:09.851059 (+ 5102us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:54477: BlockingRecv error: recv got EOF from 127.8.29.193:54477 (error 108)
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":277,"threads_started":1}
Sep 23 17:29:09 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:09 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:09.861006 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:09.852857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:52189)
0923 17:29:09.853110 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:09.853128 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:09.853235 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:09.853602 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:09.853605 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:09.853914 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:09.854056 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:09.854064 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.855505 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.855509 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:09.856388 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:09.856400 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.857270 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.857274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:09.857394 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:09.857926 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:09.857943 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:09.860793 (+ 2850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137}
W20260923 17:29:09.861292 17297 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:09.853110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:52189 (local address 127.8.29.254:44205)
0923 17:29:09.853297 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:09.853303 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:09.853323 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:29:09.853385 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:09.853391 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:09.853479 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:09.853590 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:09.854198 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.855360 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.856527 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.857143 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.857266 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:09.861063 (+ 3797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:52189: BlockingRecv error: recv got EOF from 127.8.29.193:52189 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 23 17:29:09 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:09 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:09.869490 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:09.861771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:57901)
0923 17:29:09.861935 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:09.861953 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:09.862060 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:09.862365 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:09.862370 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:09.862701 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:09.862855 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:09.862864 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.863864 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.863869 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:09.864788 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:09.864801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.865473 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.865478 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:09.865661 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:09.866206 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:09.866224 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:09.869169 (+ 2945us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:29:09.869877 17295 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:29:09.870126 17297 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:09.861924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:57901 (local address 127.8.29.254:44205)
0923 17:29:09.862086 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:09.862091 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:09.862110 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:29:09.862164 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:09.862169 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:09.862245 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:09.862324 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:09.862994 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.863721 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.864925 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:09.865338 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:09.865445 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:09.869437 (+ 3992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:57901: BlockingRecv error: recv got EOF from 127.8.29.193:57901 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260923 17:29:10.167979 17299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:10.168285 17299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:10.168409 17299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:10.175688 17299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:10.175813 17299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:10.175963 17299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:29:10.185899 17299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.194.keytab
--principal=oryx/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:45299
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=38593
--tserver_master_addrs=127.8.29.254:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:10.187273 17299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:10.188623 17299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:10.202116 17305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:10.202173 17304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:10.202147 17307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:10.203193 17299 server_base.cc:1034] running on GCE node
I20260923 17:29:10.203820 17299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:10.204814 17299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:10.206010 17299 hybrid_clock.cc:648] HybridClock initialized: now 1790184550205959 us; error 48 us; skew 500 ppm
I20260923 17:29:10.206317 17299 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184550, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:10.209381 17299 init.cc:377] Logged in from keytab as oryx/127.8.29.194@KRBTEST.COM (short username oryx)
I20260923 17:29:10.210947 17299 webserver.cc:460] Webserver started at http://127.8.29.194:38593/ using document root <none> and password file <none>
I20260923 17:29:10.211647 17299 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:10.211756 17299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:10.217253 17299 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:29:10.219991 17314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:10.221572 17299 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:29:10.221735 17299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "ca697bba16784b7a85d312f0229b587d"
format_stamp: "Formatted at 2026-09-23 17:29:07 on dist-test-slave-p7jt"
server_key: "d8eeae48e5bb675c5001cf399a537f9c"
server_key_iv: "86addcf3bf40bb82a29ccae78e655337"
server_key_version: "encryptionkey@0"
I20260923 17:29:10.222774 17299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:10.248198 17299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:10.252380 17299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:10.252645 17299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:10.253718 17299 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:10.255133 17299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:10.255241 17299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:10.255348 17299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:10.255411 17299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:10.286919 17299 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:45299
I20260923 17:29:10.286998 17426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:45299 every 8 connection(s)
I20260923 17:29:10.288411 17299 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:29:10.298266 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 17299
I20260923 17:29:10.298949 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:40371
--local_ip_for_outbound_sockets=127.8.29.195
--tserver_master_addrs=127.8.29.254:44205
--webserver_port=37519
--webserver_interface=127.8.29.195
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.29.195
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:29:10.305145 17297 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:10.290930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:54625 (local address 127.8.29.254:44205)
0923 17:29:10.291141 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:10.291147 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:10.292335 (+ 1188us) server_negotiation.cc:407] Connection header received
0923 17:29:10.294206 (+ 1871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:10.294211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:10.294292 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:10.294398 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:10.296019 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.296788 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.298681 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.299226 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.299328 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.304950 (+ 5622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:54625: BlockingRecv error: recv got EOF from 127.8.29.194:54625 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:29:10.306061 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:10.291469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:54625)
0923 17:29:10.292176 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:10.292216 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:10.293892 (+ 1676us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:10.294650 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:10.294663 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:10.295203 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:10.295790 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.295808 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.296943 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.296946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.298509 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.298524 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.299474 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.299481 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.299787 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.300519 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:10.300545 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:10.304779 (+ 4234us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":270,"threads_started":1}
W20260923 17:29:10.306761 17427 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:44205 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.29.254:44205: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:10.315702 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:10.307354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:34425)
0923 17:29:10.307541 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:10.307557 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:10.307650 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:10.307889 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:10.307892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:10.308206 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:10.308344 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.308352 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.309681 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.309685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.310677 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.310691 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.311657 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.311664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.311806 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.312376 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:10.312393 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:10.315467 (+ 3074us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:29:10.315989 17297 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:10.307451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:34425 (local address 127.8.29.254:44205)
0923 17:29:10.307652 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:10.307657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:10.307679 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:29:10.307735 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:10.307739 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:10.307798 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:10.307875 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:10.308487 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.309514 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.310819 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.311487 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.311629 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.315761 (+ 4132us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:34425: BlockingRecv error: recv got EOF from 127.8.29.194:34425 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:10.324144 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:10.316489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:53841)
0923 17:29:10.316644 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:10.316662 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:10.316772 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:10.317100 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:10.317104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:10.317423 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:10.317576 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.317585 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.318587 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.318592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.319503 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.319516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.320208 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.320214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.320373 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.320957 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:10.320975 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:10.323850 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 17:29:10.324447 17427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:29:10.324558 17297 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:10.316647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:53841 (local address 127.8.29.254:44205)
0923 17:29:10.316834 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:10.316840 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:10.316858 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:29:10.316915 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:10.316919 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:10.316974 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:10.317054 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:10.317725 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.318443 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.319643 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.320062 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.320172 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.324085 (+ 3913us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:53841: BlockingRecv error: recv got EOF from 127.8.29.194:53841 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 17:29:10.630964 17430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:10.631294 17430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:10.631422 17430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:10.638772 17430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:10.638909 17430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:10.639056 17430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:29:10.648939 17430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/krb5kdc/oryx_127.8.29.195.keytab
--principal=oryx/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:40371
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=37519
--tserver_master_addrs=127.8.29.254:44205
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:10.650296 17430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:10.651610 17430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:10.664357 17435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:10.664377 17436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:10.664621 17438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:10.665010 17430 server_base.cc:1034] running on GCE node
I20260923 17:29:10.665649 17430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:10.666657 17430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:10.667876 17430 hybrid_clock.cc:648] HybridClock initialized: now 1790184550667826 us; error 41 us; skew 500 ppm
I20260923 17:29:10.668219 17430 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184550, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:10.671283 17430 init.cc:377] Logged in from keytab as oryx/127.8.29.195@KRBTEST.COM (short username oryx)
I20260923 17:29:10.672747 17430 webserver.cc:460] Webserver started at http://127.8.29.195:37519/ using document root <none> and password file <none>
I20260923 17:29:10.673454 17430 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:10.673583 17430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:10.679127 17430 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:29:10.681965 17445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:10.683472 17430 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:29:10.683632 17430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "845d3f7d9e6143b791c3ebcd5129f2d8"
format_stamp: "Formatted at 2026-09-23 17:29:08 on dist-test-slave-p7jt"
server_key: "eddedf30f12909fa8b2f27e1bf03f250"
server_key_iv: "2c41fac05cbc4ca3778b4186885a38f8"
server_key_version: "encryptionkey@0"
I20260923 17:29:10.684582 17430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:10.696952 17430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:10.701078 17430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:10.701341 17430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:10.702291 17430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:10.703609 17430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:10.703701 17430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:10.703790 17430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:10.703860 17430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:10.733229 17430 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:40371
I20260923 17:29:10.733297 17557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:40371 every 8 connection(s)
I20260923 17:29:10.734670 17430 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:29:10.735301 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 17430
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:10.752731 17297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:10.736231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.8.29.254:44205)
0923 17:29:10.736429 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:10.736435 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:10.736857 (+ 422us) server_negotiation.cc:407] Connection header received
0923 17:29:10.737034 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:10.737038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:10.737106 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:10.737181 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:10.739159 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.740349 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.742023 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.742701 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.742832 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.745465 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:10.745478 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:10.745494 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:10.745541 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:10.749226 (+ 3685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:10.749949 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:10.749956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:10.749964 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:10.750180 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:10.750632 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:10.750637 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:10.750640 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:10.751221 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:10.751582 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:10.751985 (+ 403us) server_negotiation.cc:299] Negotiation successful
0923 17:29:10.752355 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
W20260923 17:29:10.754914 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:10.737584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:35801 (local address 127.8.29.254:44205)
0923 17:29:10.741926 (+ 4342us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:10.741931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:10.741960 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:29:10.743670 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:10.743675 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:10.743755 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:10.743866 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:10.745500 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.746395 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.748690 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.749186 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.749285 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.754698 (+ 5413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:35801: BlockingRecv error: recv got EOF from 127.8.29.195:35801 (error 108)
Metrics: {"server-negotiator.queue_time_us":4237,"thread_start_us":4130,"threads_started":1}
I20260923 17:29:10.755681 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:10.738261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:35801)
0923 17:29:10.741154 (+ 2893us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:10.741212 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:10.743401 (+ 2189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:10.744135 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:10.744144 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:10.744690 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:10.745297 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.745322 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.746569 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.746574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.748532 (+ 1958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.748549 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.749413 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.749421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.749630 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.750200 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:10.750229 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:10.754444 (+ 4215us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2515,"thread_start_us":2317,"threads_started":1}
W20260923 17:29:10.756429 17558 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:44205 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.29.254:44205: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:10.764626 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:10.757184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:60283)
0923 17:29:10.757344 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:10.757359 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:10.757443 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:10.757836 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:10.757840 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:10.758186 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:10.758336 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.758344 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.759386 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.759390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.760368 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.760383 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.761083 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.761088 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.761224 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.761779 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:10.761797 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:10.764426 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260923 17:29:10.765112 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:10.757167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:60283 (local address 127.8.29.254:44205)
0923 17:29:10.757598 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:10.757602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:10.757614 (+ 12us) server_negotiation.cc:407] Connection header received
0923 17:29:10.757648 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:10.757652 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:10.757701 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:10.757768 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:10.758477 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.759236 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.760530 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.760932 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.761030 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.764632 (+ 3602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:60283: BlockingRecv error: recv got EOF from 127.8.29.195:60283 (error 108)
Metrics: {"server-negotiator.queue_time_us":329}
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:10.772884 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:10.765235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:51727)
0923 17:29:10.765421 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:10.765436 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:10.765534 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:10.765830 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:10.765834 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:10.766136 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:10.766276 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.766283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.767275 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.767279 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.768316 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.768330 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.769171 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.769176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.769334 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.769892 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:10.769912 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:10.772655 (+ 2743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87}
W20260923 17:29:10.773218 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:10.765344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:51727 (local address 127.8.29.254:44205)
0923 17:29:10.765596 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:10.765602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:10.765619 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:29:10.765661 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:10.765664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:10.765716 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:10.765793 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:10.766404 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.767146 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.768486 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.769028 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.769163 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.772907 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:51727: BlockingRecv error: recv got EOF from 127.8.29.195:51727 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
W20260923 17:29:10.773258 17558 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:10 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:10.879439 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:10.870671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:48707)
0923 17:29:10.871244 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:10.871260 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:10.871384 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:10.871849 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:10.871853 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:10.872357 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:10.872572 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.872583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.873608 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.873613 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.874651 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:10.874665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.875376 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.875381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:10.875512 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.876114 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:10.876132 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:10.879142 (+ 3010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":336,"threads_started":1}
W20260923 17:29:10.879669 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:10.870816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:48707 (local address 127.8.29.254:44205)
0923 17:29:10.870975 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:10.870979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:10.871364 (+ 385us) server_negotiation.cc:407] Connection header received
0923 17:29:10.871602 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:10.871606 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:10.871682 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:10.871763 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:10.872744 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.873452 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.874809 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:10.875206 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:10.875305 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:10.879432 (+ 4127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:48707: BlockingRecv error: recv got EOF from 127.8.29.193:48707 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 17:29:11 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:11 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:11.333492 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:11.325136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:40715)
0923 17:29:11.325675 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:11.325695 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:11.325849 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:11.326222 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:11.326227 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:11.326805 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:11.327058 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:11.327072 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.328258 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.328262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:11.329231 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:11.329244 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.329900 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.329904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:11.330029 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:11.330607 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:11.330625 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:11.333272 (+ 2647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":291,"threads_started":1}
W20260923 17:29:11.333658 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:11.325240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:40715 (local address 127.8.29.254:44205)
0923 17:29:11.325440 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:11.325446 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:11.325793 (+ 347us) server_negotiation.cc:407] Connection header received
0923 17:29:11.325977 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:11.325981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:11.326057 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:11.326150 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:11.327210 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.328138 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.329344 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.329764 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.329849 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:11.333460 (+ 3611us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:40715: BlockingRecv error: recv got EOF from 127.8.29.194:40715 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 17:29:11 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:11 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:11.782140 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:11.773941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:57517)
0923 17:29:11.774403 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:11.774419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:11.774593 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:11.774916 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:11.774921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:11.775379 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:11.775575 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:11.775585 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.776605 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.776609 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:11.777559 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:11.777571 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.778346 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.778353 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:11.778532 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:11.779074 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:11.779090 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:11.781900 (+ 2810us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":244,"threads_started":1}
W20260923 17:29:11.782438 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:11.773992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:57517 (local address 127.8.29.254:44205)
0923 17:29:11.774166 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:11.774172 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:11.774526 (+ 354us) server_negotiation.cc:407] Connection header received
0923 17:29:11.774722 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:11.774726 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:11.774789 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:11.774873 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:11.775715 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.776464 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.777695 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.778144 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.778253 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:11.782188 (+ 3935us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:57517: BlockingRecv error: recv got EOF from 127.8.29.195:57517 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 23 17:29:11 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:11 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:11.890885 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:11.880626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:59159)
0923 17:29:11.881154 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:11.881170 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:11.881300 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:11.881755 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:11.881759 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:11.882184 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:11.882332 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:11.882342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.883458 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.883463 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:11.885001 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:11.885030 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.886000 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.886007 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:11.886224 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:11.886836 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:11.886860 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:11.890553 (+ 3693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":309,"threads_started":1}
W20260923 17:29:11.891049 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:11.880757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:59159 (local address 127.8.29.254:44205)
0923 17:29:11.880976 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:11.880981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:11.881245 (+ 264us) server_negotiation.cc:407] Connection header received
0923 17:29:11.881416 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:11.881430 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:11.881519 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:11.881611 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:11.882538 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.883298 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.885181 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:11.885756 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:11.885894 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:11.890780 (+ 4886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:59159: BlockingRecv error: recv got EOF from 127.8.29.193:59159 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 23 17:29:12 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:12 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:12.342370 17567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:12.334491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:44333)
0923 17:29:12.334993 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:12.335009 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:12.335128 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:12.335440 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:12.335444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:12.335848 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:12.335996 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:12.336005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.336975 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.336980 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:12.337899 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:12.337912 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.338573 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.338577 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:12.338701 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:12.339276 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:12.339294 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:12.342129 (+ 2835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":287,"threads_started":1}
W20260923 17:29:12.342545 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:12.334580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:44333 (local address 127.8.29.254:44205)
0923 17:29:12.334792 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:12.334799 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:12.335112 (+ 313us) server_negotiation.cc:407] Connection header received
0923 17:29:12.335212 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:12.335215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:12.335277 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:12.335365 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:12.336152 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.336845 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.338011 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.338433 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.338534 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:12.342329 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:44333: BlockingRecv error: recv got EOF from 127.8.29.194:44333 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 17:29:12 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:12 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:12.796619 17568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:12.783101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:44969)
0923 17:29:12.783628 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:12.783648 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:12.783776 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:12.784150 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:12.784154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:12.784539 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:12.784696 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:12.784705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.785699 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.785704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:12.786633 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:12.786645 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.787354 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.787359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:12.787511 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:12.793071 (+ 5560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:12.793102 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:12.796372 (+ 3270us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":286,"threads_started":1}
W20260923 17:29:12.796838 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:12.783187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:44969 (local address 127.8.29.254:44205)
0923 17:29:12.783423 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:12.783430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:12.783729 (+ 299us) server_negotiation.cc:407] Connection header received
0923 17:29:12.783895 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:12.783900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:12.783980 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:12.784059 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:12.784835 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.785571 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.786763 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.787206 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.787308 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:12.796615 (+ 9307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:44969: BlockingRecv error: recv got EOF from 127.8.29.195:44969 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 17:29:12 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:12 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:12.900167 17569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:12.891958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:52571)
0923 17:29:12.892526 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:12.892542 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:12.892696 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:12.893054 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:12.893059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:12.893531 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:12.893735 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:12.893748 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.894929 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.894933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:12.895819 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:12.895830 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.896490 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.896494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:12.896612 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:12.897172 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:12.897189 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:12.899840 (+ 2651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":303,"threads_started":1}
W20260923 17:29:12.900314 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:12.892079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:52571 (local address 127.8.29.254:44205)
0923 17:29:12.892300 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:12.892308 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:12.892636 (+ 328us) server_negotiation.cc:407] Connection header received
0923 17:29:12.892816 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:12.892820 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:12.892890 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:12.893038 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:12.893900 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.894807 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.895970 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:12.896373 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:12.896468 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:12.900094 (+ 3626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:52571: BlockingRecv error: recv got EOF from 127.8.29.193:52571 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 17:29:13 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:13 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:13.352057 17570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:13.343425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:57513)
0923 17:29:13.343913 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:13.343929 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:13.344042 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:13.344336 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:13.344339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:13.344672 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:13.344836 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:13.344845 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.345996 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.346000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:13.347027 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:13.347043 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.347774 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.347779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:13.347955 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:13.348582 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:13.348600 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:13.351725 (+ 3125us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":259,"threads_started":1}
W20260923 17:29:13.352243 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:13.343563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:57513 (local address 127.8.29.254:44205)
0923 17:29:13.343743 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:13.343748 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:13.344034 (+ 286us) server_negotiation.cc:407] Connection header received
0923 17:29:13.344138 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:13.344143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:13.344216 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:13.344310 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:13.345005 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.345802 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.347194 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.347610 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.347709 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:13.351997 (+ 4288us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:57513: BlockingRecv error: recv got EOF from 127.8.29.194:57513 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 23 17:29:13 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:13 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:13.806046 17571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:13.797676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:43853)
0923 17:29:13.798550 (+ 874us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:13.798572 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:13.798706 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:13.799111 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:13.799116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:13.799480 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:13.799647 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:13.799657 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.800678 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.800682 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:13.801585 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:13.801597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.802232 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.802237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:13.802371 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:13.802949 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:13.802966 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:13.805788 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":622,"threads_started":1}
W20260923 17:29:13.806238 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:13.797746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:43853 (local address 127.8.29.254:44205)
0923 17:29:13.797921 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:13.797926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:13.798662 (+ 736us) server_negotiation.cc:407] Connection header received
0923 17:29:13.798821 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:13.798826 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:13.798929 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:13.799036 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:13.799793 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.800546 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.801695 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.802102 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.802188 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:13.806033 (+ 3845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:43853: BlockingRecv error: recv got EOF from 127.8.29.195:43853 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 17:29:13 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:13 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:13.909403 17572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:13.901167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:41477)
0923 17:29:13.901678 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:13.901694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:13.901809 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:13.902165 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:13.902168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:13.902572 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:13.902734 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:13.902744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.903739 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.903743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:13.904691 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:13.904703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.905383 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.905387 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:13.905513 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:13.906077 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:13.906095 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:13.909132 (+ 3037us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":290,"threads_started":1}
W20260923 17:29:13.909767 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:13.901297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:41477 (local address 127.8.29.254:44205)
0923 17:29:13.901470 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:13.901476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:13.901761 (+ 285us) server_negotiation.cc:407] Connection header received
0923 17:29:13.901904 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:13.901908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:13.901970 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:13.902062 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:13.902878 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.903638 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.904824 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:13.905270 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:13.905365 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:13.909459 (+ 4094us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:41477: BlockingRecv error: recv got EOF from 127.8.29.193:41477 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 23 17:29:14 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:14 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:14.362069 17573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:14.353311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:50671)
0923 17:29:14.353901 (+ 590us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:14.353919 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:14.354078 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:14.354417 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:14.354421 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:14.354836 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:14.354986 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:14.354996 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.356205 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.356212 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:14.357734 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:14.357751 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.358497 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.358504 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:14.358663 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:14.359194 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:14.359211 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:14.361848 (+ 2637us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":380,"threads_started":1}
W20260923 17:29:14.362289 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:14.353387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:50671 (local address 127.8.29.254:44205)
0923 17:29:14.353593 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:14.353599 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:14.354021 (+ 422us) server_negotiation.cc:407] Connection header received
0923 17:29:14.354204 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:14.354208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:14.354271 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:14.354352 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:14.355138 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.356064 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.357888 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.358330 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.358422 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:14.362063 (+ 3641us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:50671: BlockingRecv error: recv got EOF from 127.8.29.194:50671 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 23 17:29:14 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:14 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:14.815452 17574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:14.806974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:59121)
0923 17:29:14.807466 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:14.807482 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:14.807609 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:14.807932 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:14.807935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:14.808266 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:14.808412 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:14.808420 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.809419 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.809423 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:14.810408 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:14.810423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.811234 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.811239 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:14.811381 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:14.811965 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:14.811984 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:14.815113 (+ 3129us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":283,"threads_started":1}
W20260923 17:29:14.815594 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:14.807051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:59121 (local address 127.8.29.254:44205)
0923 17:29:14.807203 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:14.807208 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:14.807581 (+ 373us) server_negotiation.cc:407] Connection header received
0923 17:29:14.807709 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:14.807713 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:14.807805 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:14.807891 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:14.808542 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.809289 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.810583 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.811094 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.811189 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:14.815342 (+ 4153us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:59121: BlockingRecv error: recv got EOF from 127.8.29.195:59121 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 23 17:29:14 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:14 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:14.920321 17575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:14.910683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:36637)
0923 17:29:14.911258 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:14.911275 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:14.911402 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:14.911777 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:14.911781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:14.912231 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:14.912409 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:14.912420 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.913644 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.913649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:14.914776 (+ 1127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:14.914791 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.915708 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.915713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:14.915892 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:14.916720 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:14.916747 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:14.920030 (+ 3283us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":262,"threads_started":1}
W20260923 17:29:14.920648 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:14.910855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:36637 (local address 127.8.29.254:44205)
0923 17:29:14.911112 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:14.911119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:14.911382 (+ 263us) server_negotiation.cc:407] Connection header received
0923 17:29:14.911513 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:14.911517 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:14.911600 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:14.911692 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:14.912553 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.913497 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.914955 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:14.915500 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:14.915667 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:14.920314 (+ 4647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:36637: BlockingRecv error: recv got EOF from 127.8.29.193:36637 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Sep 23 17:29:15 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:15 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:15.371073 17576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.363097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:34871)
0923 17:29:15.363606 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.363623 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.363739 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.364080 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.364084 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.364452 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.364626 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.364637 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.365699 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.365703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.366669 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.366690 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.367386 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.367390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.367511 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.368061 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.368079 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.370771 (+ 2692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":238,"threads_started":1}
W20260923 17:29:15.371204 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:15.363331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:34871 (local address 127.8.29.254:44205)
0923 17:29:15.363542 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.363547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.363723 (+ 176us) server_negotiation.cc:407] Connection header received
0923 17:29:15.363834 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.363838 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.363901 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.363988 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.364800 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.365565 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.366819 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.367252 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.367350 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.371007 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:34871: BlockingRecv error: recv got EOF from 127.8.29.194:34871 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 23 17:29:15 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:15 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:15.825070 17577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.816675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:47051)
0923 17:29:15.817160 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.817176 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.817291 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.817664 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.817667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.818018 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.818210 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.818220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.819277 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.819281 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.820218 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.820230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.821120 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.821126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.821301 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.821927 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.821951 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.824721 (+ 2770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":223,"threads_started":1}
W20260923 17:29:15.825169 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:15.816767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:47051 (local address 127.8.29.254:44205)
0923 17:29:15.816945 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.816949 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.817244 (+ 295us) server_negotiation.cc:407] Connection header received
0923 17:29:15.817438 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.817443 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.817529 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.817627 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.818353 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.819143 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.820380 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.820923 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.821071 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.824933 (+ 3862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:47051: BlockingRecv error: recv got EOF from 127.8.29.195:47051 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 17:29:15 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:15 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:15.929303 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:15.921324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:49823)
0923 17:29:15.921842 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:15.921857 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:15.921984 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:15.922320 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:15.922325 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:15.922752 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:15.922903 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.922912 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.923897 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.923901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.924787 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:15.924798 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.925558 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.925564 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:15.925730 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.926319 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:15.926337 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:15.929077 (+ 2740us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":295,"threads_started":1}
W20260923 17:29:15.929538 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:15.921396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:49823 (local address 127.8.29.254:44205)
0923 17:29:15.921596 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:15.921603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:15.921945 (+ 342us) server_negotiation.cc:407] Connection header received
0923 17:29:15.922120 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:15.922125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:15.922193 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:15.922271 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:15.923050 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.923757 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.924919 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:15.925395 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:15.925510 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:15.929305 (+ 3795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:49823: BlockingRecv error: recv got EOF from 127.8.29.193:49823 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 23 17:29:16 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:16 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:16.379684 17579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:16.372023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:41333)
0923 17:29:16.372514 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:16.372531 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:16.372672 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:16.373050 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:16.373053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:16.373421 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:16.373587 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:16.373596 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.374551 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.374555 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:16.375435 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:16.375446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.376092 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.376096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:16.376213 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:16.376795 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:16.376812 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:16.379447 (+ 2635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":261,"threads_started":1}
W20260923 17:29:16.379877 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.372122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:41333 (local address 127.8.29.254:44205)
0923 17:29:16.372301 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.372305 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.372617 (+ 312us) server_negotiation.cc:407] Connection header received
0923 17:29:16.372827 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.372832 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.372926 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.373031 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.373730 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.374431 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.375546 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.375973 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.376064 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:16.379644 (+ 3580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:41333: BlockingRecv error: recv got EOF from 127.8.29.194:41333 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 23 17:29:16 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:16 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:16.835598 17580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:16.826190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:44545)
0923 17:29:16.826746 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:16.826763 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:16.826961 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:16.827390 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:16.827393 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:16.827836 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:16.828023 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:16.828036 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.829407 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.829413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:16.830323 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:16.830336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.831244 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.831250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:16.831426 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:16.832001 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:16.832021 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:16.835278 (+ 3257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":331,"threads_started":1}
W20260923 17:29:16.835786 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.826278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:44545 (local address 127.8.29.254:44205)
0923 17:29:16.826479 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.826487 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.826893 (+ 406us) server_negotiation.cc:407] Connection header received
0923 17:29:16.827155 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.827159 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.827244 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.827339 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.828174 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.829250 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.830507 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.831070 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.831193 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:16.835528 (+ 4335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:44545: BlockingRecv error: recv got EOF from 127.8.29.195:44545 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 23 17:29:16 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:16 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:16.939385 17581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:16.930283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:45025)
0923 17:29:16.931029 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:16.931045 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:16.931198 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:16.931685 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:16.931690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:16.932235 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:16.932426 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:16.932438 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.933564 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.933568 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:16.934611 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:16.934627 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.935357 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.935362 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:16.935500 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:16.936074 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:16.936092 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:16.939031 (+ 2939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":409,"threads_started":1}
W20260923 17:29:16.939513 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:16.930408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:45025 (local address 127.8.29.254:44205)
0923 17:29:16.930769 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:16.930775 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:16.931155 (+ 380us) server_negotiation.cc:407] Connection header received
0923 17:29:16.931384 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:16.931390 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:16.931478 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:16.931593 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:16.932591 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.933390 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.934756 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:16.935182 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:16.935284 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:16.939291 (+ 4007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:45025: BlockingRecv error: recv got EOF from 127.8.29.193:45025 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 23 17:29:17 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:17 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:17.389817 17582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:17.380825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:45533)
0923 17:29:17.381528 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:17.381546 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:17.381718 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:17.382083 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:17.382087 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:17.382555 (+ 468us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:17.382739 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:17.382750 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.383905 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.383912 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:17.385003 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:17.385020 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.385903 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.385908 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:17.386026 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.386654 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:17.386672 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:17.389543 (+ 2871us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":315,"threads_started":1}
W20260923 17:29:17.390079 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:17.381025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:45533 (local address 127.8.29.254:44205)
0923 17:29:17.381334 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:17.381341 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:17.381656 (+ 315us) server_negotiation.cc:407] Connection header received
0923 17:29:17.381844 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:17.381848 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:17.381912 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:17.382057 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:17.382887 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.383704 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.385198 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.385759 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.385861 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.389795 (+ 3934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:45533: BlockingRecv error: recv got EOF from 127.8.29.194:45533 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Sep 23 17:29:17 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:17 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:17.849864 17583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:17.836629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:54121)
0923 17:29:17.837321 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:17.837337 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:17.837460 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:17.837813 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:17.837817 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:17.838256 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:17.838426 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:17.838436 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.839439 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.839444 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:17.840402 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:17.840415 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.841100 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.841104 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:17.841243 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.846200 (+ 4957us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:17.846226 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:17.849491 (+ 3265us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":367,"threads_started":1}
W20260923 17:29:17.849951 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:17.836731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:54121 (local address 127.8.29.254:44205)
0923 17:29:17.836980 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:17.836985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:17.837420 (+ 435us) server_negotiation.cc:407] Connection header received
0923 17:29:17.837551 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:17.837555 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:17.837619 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:17.837707 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:17.838572 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.839295 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.840550 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.840961 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.841058 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.849737 (+ 8679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:54121: BlockingRecv error: recv got EOF from 127.8.29.195:54121 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
Sep 23 17:29:17 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:17 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:17.949110 17584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:17.940480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:42735)
0923 17:29:17.941008 (+ 528us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:17.941049 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:17.941170 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:17.941448 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:17.941451 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:17.941834 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:17.941983 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:17.941991 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.943090 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.943095 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:17.944006 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:17.944026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.944859 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.944864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:17.944986 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.945616 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:17.945637 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:17.948794 (+ 3157us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":267,"threads_started":1}
W20260923 17:29:17.949240 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:17.940703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:42735 (local address 127.8.29.254:44205)
0923 17:29:17.940912 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:17.940919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:17.941126 (+ 207us) server_negotiation.cc:407] Connection header received
0923 17:29:17.941269 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:17.941273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:17.941334 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:17.941414 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:17.942148 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.942933 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.944172 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:17.944679 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:17.944793 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:17.949015 (+ 4222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:42735: BlockingRecv error: recv got EOF from 127.8.29.193:42735 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 23 17:29:18 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:18 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:18.398800 17585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.390785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:54285)
0923 17:29:18.391308 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.391324 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:18.391455 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:18.391768 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:18.391773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:18.392168 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:18.392316 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.392325 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.393499 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.393504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.394415 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.394427 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.395164 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.395171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.395323 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.395879 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:18.395897 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:18.398557 (+ 2660us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":261,"threads_started":1}
W20260923 17:29:18.399120 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:18.390959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:54285 (local address 127.8.29.254:44205)
0923 17:29:18.391238 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:18.391245 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:18.391404 (+ 159us) server_negotiation.cc:407] Connection header received
0923 17:29:18.391571 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:18.391577 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:18.391640 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:18.391722 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:18.392459 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.393363 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.394580 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.394986 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.395104 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.398837 (+ 3733us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:54285: BlockingRecv error: recv got EOF from 127.8.29.194:54285 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Sep 23 17:29:18 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:18 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:18.859673 17586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.850842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:39941)
0923 17:29:18.851384 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.851400 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:18.851518 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:18.851834 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:18.851837 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:18.852244 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:18.852395 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.852404 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.853378 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.853382 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.854300 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.854312 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.855120 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.855125 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.855243 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.855918 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:18.855946 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:18.859375 (+ 3429us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":313,"threads_started":1}
W20260923 17:29:18.859861 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:18.850978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:39941 (local address 127.8.29.254:44205)
0923 17:29:18.851174 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:18.851178 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:18.851462 (+ 284us) server_negotiation.cc:407] Connection header received
0923 17:29:18.851610 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:18.851614 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:18.851674 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:18.851752 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:18.852527 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.853241 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.854449 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.854945 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.855060 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.859630 (+ 4570us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:39941: BlockingRecv error: recv got EOF from 127.8.29.195:39941 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 23 17:29:18 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:18 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:18.958220 17587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:18.950052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:40783)
0923 17:29:18.950542 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:18.950560 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:18.950679 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:18.951044 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:18.951048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:18.951438 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:18.951606 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.951616 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.952628 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.952632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.953592 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:18.953604 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.954302 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.954306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:18.954431 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.954999 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:18.955015 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:18.957861 (+ 2846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":241,"threads_started":1}
W20260923 17:29:18.958516 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:18.950127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:40783 (local address 127.8.29.254:44205)
0923 17:29:18.950329 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:18.950336 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:18.950664 (+ 328us) server_negotiation.cc:407] Connection header received
0923 17:29:18.950781 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:18.950785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:18.950850 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:18.950938 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:18.951761 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.952501 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.953730 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:18.954169 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:18.954264 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:18.958115 (+ 3851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:40783: BlockingRecv error: recv got EOF from 127.8.29.193:40783 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 23 17:29:19 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:19 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:19.407662 17588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.399753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:50277)
0923 17:29:19.400251 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.400267 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:19.400384 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:19.400667 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:19.400670 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:19.401035 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:19.401210 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.401220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.402295 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.402300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.403289 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.403302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.403978 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.403982 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.404103 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.404689 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:19.404706 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:19.407410 (+ 2704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":260,"threads_started":1}
W20260923 17:29:19.407887 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.399839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:50277 (local address 127.8.29.254:44205)
0923 17:29:19.400012 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.400016 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.400336 (+ 320us) server_negotiation.cc:407] Connection header received
0923 17:29:19.400470 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.400474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.400533 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.400617 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.401353 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.402159 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.403421 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.403851 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.403942 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.407651 (+ 3709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:50277: BlockingRecv error: recv got EOF from 127.8.29.194:50277 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Sep 23 17:29:19 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:19 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:19.869712 17589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.860969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:35229)
0923 17:29:19.861672 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.861689 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:19.861839 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:19.862362 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:19.862368 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:19.862869 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:19.863039 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.863049 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.864029 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.864033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.864984 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.864997 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.865728 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.865733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.865867 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.866539 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:19.866563 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:19.869442 (+ 2879us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":485,"threads_started":1}
W20260923 17:29:19.869922 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.861153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:35229 (local address 127.8.29.254:44205)
0923 17:29:19.861359 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.861365 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.861781 (+ 416us) server_negotiation.cc:407] Connection header received
0923 17:29:19.861973 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.861979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.862060 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.862164 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.863154 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.863889 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.865128 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.865555 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.865657 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.869715 (+ 4058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:35229: BlockingRecv error: recv got EOF from 127.8.29.195:35229 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 23 17:29:19 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:19 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:19.967732 17590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:19.959382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:56401)
0923 17:29:19.959906 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:19.959923 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:19.960063 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:19.960428 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:19.960433 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:19.960812 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:19.960965 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.960975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.962093 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.962098 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.963151 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:19.963167 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.963899 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.963905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:19.964065 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.964673 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:19.964690 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:19.967409 (+ 2719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":295,"threads_started":1}
W20260923 17:29:19.967876 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:19.959605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:56401 (local address 127.8.29.254:44205)
0923 17:29:19.959858 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:19.959865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:19.960052 (+ 187us) server_negotiation.cc:407] Connection header received
0923 17:29:19.960183 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:19.960187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:19.960259 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:19.960347 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:19.961160 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.961911 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.963294 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:19.963741 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:19.963851 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:19.967650 (+ 3799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:56401: BlockingRecv error: recv got EOF from 127.8.29.193:56401 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Sep 23 17:29:20 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:20 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:20.417238 17591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:20.408610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:58253)
0923 17:29:20.409129 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:20.409144 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:20.409280 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:20.409624 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:20.409630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:20.410020 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:20.410169 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:20.410179 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.411502 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.411507 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:20.412651 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:20.412679 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.413337 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.413343 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:20.413514 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.414096 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:20.414112 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:20.416934 (+ 2822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":272,"threads_started":1}
W20260923 17:29:20.417557 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:20.408802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:58253 (local address 127.8.29.254:44205)
0923 17:29:20.409042 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:20.409048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:20.409229 (+ 181us) server_negotiation.cc:407] Connection header received
0923 17:29:20.409399 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:20.409405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:20.409478 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:20.409560 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:20.410317 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.411355 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.412798 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.413195 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.413301 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.417216 (+ 3915us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:58253: BlockingRecv error: recv got EOF from 127.8.29.194:58253 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 23 17:29:20 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:20 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:20.879732 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:20.870881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:35707)
0923 17:29:20.871393 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:20.871414 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:20.871544 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:20.871906 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:20.871910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:20.872303 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:20.872451 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:20.872460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.873786 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.873823 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:20.874897 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:20.874914 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.875636 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.875642 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:20.875833 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.876448 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:20.876466 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:20.879382 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":279,"threads_started":1}
W20260923 17:29:20.879947 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:20.871163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:35707 (local address 127.8.29.254:44205)
0923 17:29:20.871362 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:20.871366 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:20.871490 (+ 124us) server_negotiation.cc:407] Connection header received
0923 17:29:20.871667 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:20.871673 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:20.871736 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:20.871827 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:20.872648 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.873639 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.875043 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.875475 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.875604 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.879656 (+ 4052us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:35707: BlockingRecv error: recv got EOF from 127.8.29.195:35707 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 23 17:29:20 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:20 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:20.977517 17593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:20.968946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:41537)
0923 17:29:20.969476 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:20.969495 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:20.969701 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:20.970060 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:20.970065 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:20.970530 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:20.970718 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:20.970728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.971749 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.971755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:20.972775 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:20.972788 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.973579 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.973586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:20.973747 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.974338 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:20.974356 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:20.977195 (+ 2839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":273,"threads_started":1}
W20260923 17:29:20.977635 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:20.969112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:41537 (local address 127.8.29.254:44205)
0923 17:29:20.969373 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:20.969379 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:20.969585 (+ 206us) server_negotiation.cc:407] Connection header received
0923 17:29:20.969840 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:20.969846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:20.969926 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:20.970010 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:20.970873 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.971603 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.972916 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:20.973373 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:20.973545 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:20.977406 (+ 3861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:41537: BlockingRecv error: recv got EOF from 127.8.29.193:41537 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 23 17:29:21 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:21 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:21.426533 17594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:21.418288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:54269)
0923 17:29:21.418889 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:21.418907 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:21.419057 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:21.419416 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:21.419421 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:21.419960 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:21.420134 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:21.420143 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.421175 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.421181 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:21.422063 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:21.422076 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.422777 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.422783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:21.422938 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:21.423496 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:21.423513 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:21.426175 (+ 2662us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":369,"threads_started":1}
W20260923 17:29:21.426726 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:21.418402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:54269 (local address 127.8.29.254:44205)
0923 17:29:21.418664 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:21.418671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:21.419014 (+ 343us) server_negotiation.cc:407] Connection header received
0923 17:29:21.419188 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:21.419193 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:21.419272 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:21.419365 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:21.420277 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.421046 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.422198 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.422640 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.422744 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:21.426442 (+ 3698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:54269: BlockingRecv error: recv got EOF from 127.8.29.194:54269 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 23 17:29:21 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:21 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:21.895282 17595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:21.880800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:43067)
0923 17:29:21.881267 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:21.881292 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:21.881418 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:21.881732 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:21.881736 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:21.882117 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:21.882289 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:21.882298 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.883294 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.883297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:21.884205 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:21.884217 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.884859 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.884863 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:21.884974 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:21.891898 (+ 6924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:21.891953 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:21.894922 (+ 2969us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":229,"threads_started":1}
W20260923 17:29:21.895392 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:21.880923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:43067 (local address 127.8.29.254:44205)
0923 17:29:21.881188 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:21.881194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:21.881368 (+ 174us) server_negotiation.cc:407] Connection header received
0923 17:29:21.881528 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:21.881532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:21.881596 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:21.881684 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:21.882412 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.883168 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.884327 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.884745 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.884839 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:21.895180 (+ 10341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:43067: BlockingRecv error: recv got EOF from 127.8.29.195:43067 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Sep 23 17:29:21 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:21 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:21.987063 17596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:21.978437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:46489)
0923 17:29:21.979028 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:21.979043 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:21.979156 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:21.979447 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:21.979450 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:21.979854 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:21.980061 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:21.980073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.981055 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.981059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:21.981965 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:21.981976 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.982769 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.982774 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:21.982912 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:21.983491 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:21.983509 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:21.986668 (+ 3159us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":299,"threads_started":1}
W20260923 17:29:21.987172 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:21.978616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:46489 (local address 127.8.29.254:44205)
0923 17:29:21.978815 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:21.978821 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:21.979113 (+ 292us) server_negotiation.cc:407] Connection header received
0923 17:29:21.979277 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:21.979281 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:21.979344 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:21.979419 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:21.980186 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.980918 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.982095 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:21.982619 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:21.982716 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:21.986949 (+ 4233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:46489: BlockingRecv error: recv got EOF from 127.8.29.193:46489 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:29:22 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:22 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:22.435935 17597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:22.427514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:38201)
0923 17:29:22.428021 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:22.428037 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:22.428167 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:22.428539 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:22.428545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:22.429006 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:22.429214 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:22.429227 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.430517 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.430523 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:22.431450 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:22.431463 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.432142 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.432147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:22.432283 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:22.432847 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:22.432865 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:22.435696 (+ 2831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":274,"threads_started":1}
W20260923 17:29:22.436151 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:22.427634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:38201 (local address 127.8.29.254:44205)
0923 17:29:22.427796 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:22.427801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:22.428139 (+ 338us) server_negotiation.cc:407] Connection header received
0923 17:29:22.428305 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:22.428311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:22.428394 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:22.428504 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:22.429369 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.430347 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.431595 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.432012 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.432108 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:22.435906 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:38201: BlockingRecv error: recv got EOF from 127.8.29.194:38201 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 23 17:29:22 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:22 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:22.908730 17598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:22.896409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:55853)
0923 17:29:22.896956 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:22.896974 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:22.897128 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:22.897487 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:22.897492 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:22.897906 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:22.898075 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:22.898085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.899250 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.899256 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:22.900233 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:22.900247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.901194 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.901200 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:22.901368 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:22.905400 (+ 4032us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:22.905422 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:22.908378 (+ 2956us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":281,"threads_started":1}
W20260923 17:29:22.908936 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:22.896558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:55853 (local address 127.8.29.254:44205)
0923 17:29:22.896799 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:22.896806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:22.897075 (+ 269us) server_negotiation.cc:407] Connection header received
0923 17:29:22.897268 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:22.897273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:22.897340 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:22.897433 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:22.898241 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.899095 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.900388 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.900989 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.901117 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:22.908689 (+ 7572us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:55853: BlockingRecv error: recv got EOF from 127.8.29.195:55853 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Sep 23 17:29:22 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:22 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:22.997048 17599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:22.988291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:48293)
0923 17:29:22.988855 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:22.988874 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:22.989016 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:22.989347 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:22.989352 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:22.989824 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:22.990033 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:22.990048 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.991098 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.991103 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:22.992059 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:22.992071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.992872 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.992877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:22.993067 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:22.993689 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:22.993707 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:22.996697 (+ 2990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":263,"threads_started":1}
W20260923 17:29:22.997201 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:22.988367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:48293 (local address 127.8.29.254:44205)
0923 17:29:22.988612 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:22.988619 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:22.988941 (+ 322us) server_negotiation.cc:407] Connection header received
0923 17:29:22.989135 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:22.989140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:22.989210 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:22.989290 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:22.990194 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.990958 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.992229 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:22.992681 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:22.992786 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:22.996959 (+ 4173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:48293: BlockingRecv error: recv got EOF from 127.8.29.193:48293 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 23 17:29:23 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:23 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:23.445317 17600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:23.436994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:33271)
0923 17:29:23.437719 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:23.437740 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:23.437871 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:23.438325 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:23.438329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:23.438753 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:23.438922 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:23.438933 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:23.439975 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:23.439978 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:23.440923 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:23.440938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:23.441688 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:23.441692 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:23.441808 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:23.442432 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:23.442448 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:23.445033 (+ 2585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":445,"threads_started":1}
W20260923 17:29:23.445497 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:23.437880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:33271 (local address 127.8.29.254:44205)
0923 17:29:23.438048 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:23.438052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:23.438069 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:29:23.438140 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:23.438144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:23.438204 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:23.438280 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:23.439080 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:23.439844 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:23.441076 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:23.441527 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:23.441635 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:23.445284 (+ 3649us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:33271: BlockingRecv error: recv got EOF from 127.8.29.194:33271 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 17:29:23 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:23 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:23.918500 17601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:23.909809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:39145)
0923 17:29:23.910289 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:23.910304 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:23.910442 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:23.910928 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:23.910933 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:23.911444 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:23.911641 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:23.911654 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:23.912822 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:23.912828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:23.913889 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:23.913903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:23.914678 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:23.914684 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:23.914848 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:23.915404 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:23.915422 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:23.918210 (+ 2788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":231,"threads_started":1}
W20260923 17:29:23.918713 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:23.909851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:39145 (local address 127.8.29.254:44205)
0923 17:29:23.910047 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:23.910053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:23.910390 (+ 337us) server_negotiation.cc:407] Connection header received
0923 17:29:23.910598 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:23.910603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:23.910684 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:23.910807 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:23.911776 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:23.912635 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:23.914024 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:23.914493 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:23.914612 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:23.918430 (+ 3818us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:39145: BlockingRecv error: recv got EOF from 127.8.29.195:39145 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 23 17:29:23 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:23 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:24.006351 17602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:23.998040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:38509)
0923 17:29:23.998544 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:23.998565 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:23.998695 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:23.999012 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:23.999015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:23.999370 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:23.999541 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:23.999550 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.000623 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.000627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:24.001666 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:24.001683 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.002505 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.002513 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:24.002670 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:24.003249 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:24.003267 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:24.006085 (+ 2818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":253,"threads_started":1}
W20260923 17:29:24.006603 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:23.998116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:38509 (local address 127.8.29.254:44205)
0923 17:29:23.998373 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:23.998378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:23.998678 (+ 300us) server_negotiation.cc:407] Connection header received
0923 17:29:23.998808 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:23.998812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:23.998876 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:23.998963 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:23.999683 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.000428 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.001819 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.002306 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.002400 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:24.006302 (+ 3902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:38509: BlockingRecv error: recv got EOF from 127.8.29.193:38509 (error 108)
Metrics: {"server-negotiator.queue_time_us":149}
Sep 23 17:29:24 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:24 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:24.454401 17603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:24.446294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:57329)
0923 17:29:24.446782 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:24.446799 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:24.446912 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:24.447190 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:24.447193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:24.447534 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:24.447734 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:24.447744 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.448759 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.448763 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:24.449683 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:24.449696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.450500 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.450506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:24.450649 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:24.451272 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:24.451290 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:24.454145 (+ 2855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":260,"threads_started":1}
W20260923 17:29:24.454662 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:24.446524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:57329 (local address 127.8.29.254:44205)
0923 17:29:24.446693 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:24.446698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:24.446890 (+ 192us) server_negotiation.cc:407] Connection header received
0923 17:29:24.446997 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:24.447001 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:24.447077 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:24.447158 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:24.447874 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.448620 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.449833 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.450346 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.450493 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:24.454384 (+ 3891us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:57329: BlockingRecv error: recv got EOF from 127.8.29.194:57329 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 23 17:29:24 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:24 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.195@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:24.927983 17604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:24.919490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.195:49995)
0923 17:29:24.920043 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:24.920060 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:24.920185 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:24.920606 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:24.920611 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:24.921009 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:24.921206 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:24.921216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.922214 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.922219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:24.923206 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:24.923220 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.923951 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.923956 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:24.924105 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:24.924703 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:24.924723 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:24.927658 (+ 2935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":228,"threads_started":1}
W20260923 17:29:24.928196 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:24.919538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:49995 (local address 127.8.29.254:44205)
0923 17:29:24.919787 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:24.919794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:24.920173 (+ 379us) server_negotiation.cc:407] Connection header received
0923 17:29:24.920355 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:24.920359 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:24.920425 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:24.920509 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:24.921338 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.922073 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.923356 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:24.923798 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:24.923894 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:24.927912 (+ 4018us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.195:49995: BlockingRecv error: recv got EOF from 127.8.29.195:49995 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
Sep 23 17:29:25 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:25 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.193@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:25.016311 17605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:25.007469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.193:53945)
0923 17:29:25.008007 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:25.008023 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:25.008158 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:25.008513 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:25.008518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:25.008944 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:25.009138 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:25.009148 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.010176 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.010180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:25.011339 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:25.011354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.012286 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.012293 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:25.012470 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:25.013088 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:25.013139 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:25.016055 (+ 2916us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":272,"threads_started":1}
W20260923 17:29:25.016566 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:25.007633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:53945 (local address 127.8.29.254:44205)
0923 17:29:25.007840 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:25.007846 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:25.008109 (+ 263us) server_negotiation.cc:407] Connection header received
0923 17:29:25.008295 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:25.008299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:25.008363 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:25.008469 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:25.009288 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.010026 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.011498 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.012107 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.012267 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:25.016305 (+ 4038us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.193:53945: BlockingRecv error: recv got EOF from 127.8.29.193:53945 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 17:29:25 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
Sep 23 17:29:25 dist-test-slave-p7jt krb5kdc[16561](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.29.194@KRBTEST.COM for oryx/127.8.29.254@KRBTEST.COM, Server not found in Kerberos database
I20260923 17:29:25.464080 17606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:25.455642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44205 (local address 127.8.29.194:57551)
0923 17:29:25.456241 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:25.456257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:25.456374 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:25.456674 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:25.456677 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:25.457101 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:25.457291 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:25.457300 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.458329 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.458333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:25.459302 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:25.459315 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.460105 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.460110 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:25.460253 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:25.460935 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:25.460956 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:25.463831 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.29.254:44205: Server oryx/127.8.29.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":400,"threads_started":1}
W20260923 17:29:25.464262 17561 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:25.455716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:57551 (local address 127.8.29.254:44205)
0923 17:29:25.455923 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:25.455931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:25.456334 (+ 403us) server_negotiation.cc:407] Connection header received
0923 17:29:25.456466 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:25.456470 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:25.456546 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:25.456626 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:25.457410 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.458191 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.459459 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.459909 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.460058 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:25.464045 (+ 3987us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.29.194:57551: BlockingRecv error: recv got EOF from 127.8.29.194:57551 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:29:25.738837 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17148
I20260923 17:29:25.750989 17608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:25.739983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.8.29.193:40951)
0923 17:29:25.740510 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:25.740517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:25.740545 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:29:25.740635 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:25.740642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:25.740878 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:25.741029 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:25.742513 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.743477 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.744662 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.745257 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.745368 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:25.747846 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:25.747855 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:25.747882 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:25.747917 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:25.750418 (+ 2501us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.29.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 17:29:25.750756 (+ 338us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38002: Request ticket server kudu/127.8.29.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":234,"threads_started":1}
W20260923 17:29:25.751205 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17148: Not authorized: Client connection negotiation failed: client connection to 127.8.29.193:40951: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.29.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 17:29:25.751371 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 17148
I20260923 17:29:25.765547 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17299
I20260923 17:29:25.777686 17609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:25.766628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.8.29.194:45299)
0923 17:29:25.767185 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:25.767192 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:25.767221 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:29:25.767279 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:25.767286 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:25.767530 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:25.767692 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:25.768945 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.769979 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.771240 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.771814 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.771912 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:25.774557 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:25.774570 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:25.774587 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:25.774630 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:25.777154 (+ 2524us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.29.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 17:29:25.777440 (+ 286us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53194: Request ticket server kudu/127.8.29.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":267,"threads_started":1}
W20260923 17:29:25.777793 17609 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53194: Request ticket server kudu/127.8.29.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 17:29:25.777930 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17299: Not authorized: Client connection negotiation failed: client connection to 127.8.29.194:45299: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.29.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 17:29:25.778105 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 17299
I20260923 17:29:25.792317 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17430
I20260923 17:29:25.804344 17610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:25.793284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56034 (local address 127.8.29.195:40371)
0923 17:29:25.793815 (+ 531us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:25.793826 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:25.793851 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:29:25.793916 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:25.793923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:25.794180 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:25.794338 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:25.795698 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.796767 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.797959 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:25.798609 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:25.798708 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:25.801204 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:25.801214 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:25.801230 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:25.801274 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:25.803819 (+ 2545us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.29.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0923 17:29:25.804085 (+ 266us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56034: Request ticket server kudu/127.8.29.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":282,"threads_started":1}
W20260923 17:29:25.804458 17610 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56034: Request ticket server kudu/127.8.29.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260923 17:29:25.804550 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17430: Not authorized: Client connection negotiation failed: client connection to 127.8.29.195:40371: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.8.29.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260923 17:29:25.804709 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 17430
I20260923 17:29:25.817823 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17079
I20260923 17:29:25.819093 17141 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:25.934612 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 17079
2026-09-23T17:29:25Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (20961 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T17:29:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:25Z Disabled control of system clock
I20260923 17:29:26.005941 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:41871
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:41871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 17:29:26.332413 17615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:26.332742 17615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:26.332864 17615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:26.340262 17615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:26.340384 17615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:26.340425 17615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:26.340502 17615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:26.340556 17615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:26.350047 17615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:41871
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:41871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:26.351423 17615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:26.353223 17615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:26.365118 17620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:26.365175 17621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:26.365144 17623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:26.366003 17615 server_base.cc:1034] running on GCE node
I20260923 17:29:26.366878 17615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:26.368307 17615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:26.369505 17615 hybrid_clock.cc:648] HybridClock initialized: now 1790184566369462 us; error 38 us; skew 500 ppm
I20260923 17:29:26.369828 17615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:29:26.372011 17615 webserver.cc:460] Webserver started at http://127.8.29.254:44145/ using document root <none> and password file <none>
I20260923 17:29:26.372700 17615 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:26.372789 17615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:26.373063 17615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:26.375993 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "285cda4a8c3f479daae10102e7042fae"
format_stamp: "Formatted at 2026-09-23 17:29:26 on dist-test-slave-p7jt"
server_key: "fafeb1d9ad60447941aef10e397922b9"
server_key_iv: "545100a21cb451d1652ab1110fcdcc21"
server_key_version: "encryptionkey@0"
I20260923 17:29:26.376781 17615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "285cda4a8c3f479daae10102e7042fae"
format_stamp: "Formatted at 2026-09-23 17:29:26 on dist-test-slave-p7jt"
server_key: "fafeb1d9ad60447941aef10e397922b9"
server_key_iv: "545100a21cb451d1652ab1110fcdcc21"
server_key_version: "encryptionkey@0"
I20260923 17:29:26.381935 17615 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 17:29:26.385397 17629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:26.386960 17615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:29:26.387121 17615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "285cda4a8c3f479daae10102e7042fae"
format_stamp: "Formatted at 2026-09-23 17:29:26 on dist-test-slave-p7jt"
server_key: "fafeb1d9ad60447941aef10e397922b9"
server_key_iv: "545100a21cb451d1652ab1110fcdcc21"
server_key_version: "encryptionkey@0"
I20260923 17:29:26.387332 17615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:26.417035 17615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:26.417961 17615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:26.418205 17615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:26.435947 17615 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:41871
I20260923 17:29:26.435954 17680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:41871 every 8 connection(s)
I20260923 17:29:26.437356 17615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:26.438011 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 17615
I20260923 17:29:26.438249 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:29:26.438987 8311 external_mini_cluster.cc:1428] Setting key d0d49bf3874a6e536b84db2413530893
I20260923 17:29:26.441847 17681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:26.452979 17684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:26.441090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43812 (local address 127.8.29.254:41871)
0923 17:29:26.442108 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:26.442123 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:26.442169 (+ 46us) server_negotiation.cc:407] Connection header received
0923 17:29:26.443783 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:26.443803 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:26.444292 (+ 489us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:26.444638 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:26.445276 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:26.446637 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:26.448447 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:26.449181 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:26.449309 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:26.450556 (+ 1247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:26.450564 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:26.450583 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:26.451175 (+ 592us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:26.451265 (+ 90us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:26.451289 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:26.451379 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:26.451614 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 17:29:26.451964 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":329,"threads_started":1}
I20260923 17:29:26.454097 17681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae: Bootstrap starting.
I20260923 17:29:26.457298 17681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:26.458300 17681 log.cc:824] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:26.461083 17681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae: No bootstrap required, opened a new log
I20260923 17:29:26.468686 17681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285cda4a8c3f479daae10102e7042fae" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41871 } }
I20260923 17:29:26.469060 17681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:26.469135 17681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 285cda4a8c3f479daae10102e7042fae, State: Initialized, Role: FOLLOWER
I20260923 17:29:26.469646 17681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285cda4a8c3f479daae10102e7042fae" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41871 } }
I20260923 17:29:26.469839 17681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:26.469930 17681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:26.470068 17681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:26.472173 17681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285cda4a8c3f479daae10102e7042fae" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41871 } }
I20260923 17:29:26.472643 17681 leader_election.cc:302] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 285cda4a8c3f479daae10102e7042fae; no voters:
I20260923 17:29:26.473264 17681 leader_election.cc:288] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:26.473393 17686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:26.474445 17686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [term 1 LEADER]: Becoming Leader. State: Replica: 285cda4a8c3f479daae10102e7042fae, State: Running, Role: LEADER
I20260923 17:29:26.475129 17686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285cda4a8c3f479daae10102e7042fae" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41871 } }
I20260923 17:29:26.475477 17681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:26.477653 17687 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "285cda4a8c3f479daae10102e7042fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285cda4a8c3f479daae10102e7042fae" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41871 } } }
I20260923 17:29:26.477955 17687 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:26.478219 17688 sys_catalog.cc:455] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 285cda4a8c3f479daae10102e7042fae. Latest consensus state: current_term: 1 leader_uuid: "285cda4a8c3f479daae10102e7042fae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "285cda4a8c3f479daae10102e7042fae" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41871 } } }
I20260923 17:29:26.478613 17688 sys_catalog.cc:458] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:26.480505 17693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:26.485836 17693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:26.495409 17693 catalog_manager.cc:1347] Generated new cluster ID: 78d3e14c24184804b0eac274289e79c3
I20260923 17:29:26.495587 17693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:26.514559 17693 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:26.516428 17693 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:26.527828 17693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 285cda4a8c3f479daae10102e7042fae: Generated new TSK 0
I20260923 17:29:26.529096 17693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:26.543833 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41871
--builtin_ntp_servers=127.8.29.212:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 17:29:26.879042 17705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:26.879354 17705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:26.879508 17705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:26.887816 17705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:26.887919 17705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:26.888065 17705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:29:26.897400 17705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:26.898921 17705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:26.900751 17705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:26.913383 17713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:26.913415 17711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:26.913383 17710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:26.914292 17705 server_base.cc:1034] running on GCE node
I20260923 17:29:26.914983 17705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:26.915973 17705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:26.917181 17705 hybrid_clock.cc:648] HybridClock initialized: now 1790184566917128 us; error 39 us; skew 500 ppm
I20260923 17:29:26.917491 17705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:29:26.919999 17705 webserver.cc:460] Webserver started at http://127.8.29.193:33071/ using document root <none> and password file <none>
I20260923 17:29:26.920804 17705 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:26.920925 17705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:26.921274 17705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:26.924346 17705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "6ec091e0564d40f088b5dd9468f93f5f"
format_stamp: "Formatted at 2026-09-23 17:29:26 on dist-test-slave-p7jt"
server_key: "c494cdf19617b3780db2929fd3dbf3dd"
server_key_iv: "ce34d6158303a159fcf6bd6f382d89f4"
server_key_version: "encryptionkey@0"
I20260923 17:29:26.925153 17705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "6ec091e0564d40f088b5dd9468f93f5f"
format_stamp: "Formatted at 2026-09-23 17:29:26 on dist-test-slave-p7jt"
server_key: "c494cdf19617b3780db2929fd3dbf3dd"
server_key_iv: "ce34d6158303a159fcf6bd6f382d89f4"
server_key_version: "encryptionkey@0"
I20260923 17:29:26.930498 17705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:29:26.933981 17719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:26.935393 17705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:29:26.935552 17705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "6ec091e0564d40f088b5dd9468f93f5f"
format_stamp: "Formatted at 2026-09-23 17:29:26 on dist-test-slave-p7jt"
server_key: "c494cdf19617b3780db2929fd3dbf3dd"
server_key_iv: "ce34d6158303a159fcf6bd6f382d89f4"
server_key_version: "encryptionkey@0"
I20260923 17:29:26.935760 17705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:26.954566 17705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:26.955476 17705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:26.955770 17705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:26.956791 17705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:26.958372 17705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:26.958505 17705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:26.958616 17705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:26.958684 17705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:26.988929 17705 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:38979
I20260923 17:29:26.988981 17831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:38979 every 8 connection(s)
I20260923 17:29:26.990214 17705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:29:26.999307 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 17705
I20260923 17:29:26.999545 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:29:27.000349 8311 external_mini_cluster.cc:1428] Setting key eebee7dbbc3d99522798b8b5f9f1d9f7
I20260923 17:29:27.003585 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41871
--builtin_ntp_servers=127.8.29.212:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:29:27.003870 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:26.992877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:38983 (local address 127.8.29.254:41871)
0923 17:29:26.993331 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:26.993336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:26.994540 (+ 1204us) server_negotiation.cc:407] Connection header received
0923 17:29:26.996322 (+ 1782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:26.996328 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:26.996413 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:26.996509 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:26.997665 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:26.998396 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.000806 (+ 2410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.001287 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.001408 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:27.003028 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:27.003034 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:27.003038 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:27.003057 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:27.003098 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:27.003111 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:27.003186 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:27.003466 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 17:29:27.003557 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":256,"threads_started":1}
I20260923 17:29:27.005483 17835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:26.993437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:41871 (local address 127.8.29.193:38983)
0923 17:29:26.994355 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:26.994398 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:26.996044 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:26.996766 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:26.996776 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:26.996873 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:26.997486 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:26.997503 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:26.998565 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:26.998569 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:27.000615 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:27.000640 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.001535 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.001542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:27.001772 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:27.002591 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:29:27.002595 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:27.002659 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:29:27.002776 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:27.003215 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:27.003227 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:27.003236 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:29:27.003460 (+ 224us) client_negotiation.cc:240] Negotiation successful
0923 17:29:27.003941 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":267,"threads_started":1}
I20260923 17:29:27.007097 17832 heartbeater.cc:347] Connected to a master server at 127.8.29.254:41871
I20260923 17:29:27.007441 17832 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:27.008402 17832 heartbeater.cc:511] Master 127.8.29.254:41871 requested a full tablet report, sending...
I20260923 17:29:27.010695 17646 ts_manager.cc:194] Registered new tserver with Master: 6ec091e0564d40f088b5dd9468f93f5f (127.8.29.193:38979)
I20260923 17:29:27.012239 17646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.193:38983
W20260923 17:29:27.335776 17836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:27.336094 17836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:27.336215 17836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:27.343587 17836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:27.343703 17836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:27.343842 17836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:29:27.353058 17836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:27.354375 17836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:27.356187 17836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:27.369867 17841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:27.369867 17842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:27.370065 17844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:27.370527 17836 server_base.cc:1034] running on GCE node
I20260923 17:29:27.371223 17836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:27.372207 17836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:27.373415 17836 hybrid_clock.cc:648] HybridClock initialized: now 1790184567373360 us; error 48 us; skew 500 ppm
I20260923 17:29:27.373718 17836 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:29:27.376063 17836 webserver.cc:460] Webserver started at http://127.8.29.194:42079/ using document root <none> and password file <none>
I20260923 17:29:27.376832 17836 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:27.376957 17836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:27.377272 17836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:27.380277 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "8e1df087d7dc40968dcde345e72ec32a"
format_stamp: "Formatted at 2026-09-23 17:29:27 on dist-test-slave-p7jt"
server_key: "a1dfb115d4465589bcd8080a6f6ecedd"
server_key_iv: "4fcf2deada93f33f2f4ffdf82e2ef5a2"
server_key_version: "encryptionkey@0"
I20260923 17:29:27.381162 17836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "8e1df087d7dc40968dcde345e72ec32a"
format_stamp: "Formatted at 2026-09-23 17:29:27 on dist-test-slave-p7jt"
server_key: "a1dfb115d4465589bcd8080a6f6ecedd"
server_key_iv: "4fcf2deada93f33f2f4ffdf82e2ef5a2"
server_key_version: "encryptionkey@0"
I20260923 17:29:27.387012 17836 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:29:27.390965 17850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:27.392618 17836 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:29:27.392851 17836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "8e1df087d7dc40968dcde345e72ec32a"
format_stamp: "Formatted at 2026-09-23 17:29:27 on dist-test-slave-p7jt"
server_key: "a1dfb115d4465589bcd8080a6f6ecedd"
server_key_iv: "4fcf2deada93f33f2f4ffdf82e2ef5a2"
server_key_version: "encryptionkey@0"
I20260923 17:29:27.393086 17836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:27.408457 17836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:27.409320 17836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:27.409586 17836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:27.410634 17836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:27.412045 17836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:27.412127 17836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:27.412209 17836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:27.412261 17836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:27.443200 17836 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:44947
I20260923 17:29:27.443276 17962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:44947 every 8 connection(s)
I20260923 17:29:27.444532 17836 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:29:27.448519 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 17836
I20260923 17:29:27.448707 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:29:27.449208 8311 external_mini_cluster.cc:1428] Setting key 8bf59b3ffe6c7fa396f222204544e4f7
I20260923 17:29:27.451997 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41871
--builtin_ntp_servers=127.8.29.212:46591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 17:29:27.458724 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:27.447226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:55983 (local address 127.8.29.254:41871)
0923 17:29:27.447435 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:27.447442 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:27.449125 (+ 1683us) server_negotiation.cc:407] Connection header received
0923 17:29:27.451697 (+ 2572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:27.451702 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:27.451788 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:27.451897 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:27.453311 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.454292 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.455918 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.456446 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.456565 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:27.457925 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:27.457931 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:27.457935 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:27.457959 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:27.457999 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:27.458013 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:27.458095 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:27.458289 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 17:29:27.458396 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:29:27.460053 17965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:27.447826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:41871 (local address 127.8.29.194:55983)
0923 17:29:27.448942 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:27.449004 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:27.451363 (+ 2359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:27.452167 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:27.452179 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:27.452317 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:27.453123 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:27.453146 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.454484 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.454490 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:27.455779 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:27.455792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.456658 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.456664 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:27.456872 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:27.457583 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:29:27.457586 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:27.457633 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:29:27.457722 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:27.458100 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:27.458108 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:27.458114 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:29:27.458262 (+ 148us) client_negotiation.cc:240] Negotiation successful
0923 17:29:27.458706 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":314,"threads_started":1}
I20260923 17:29:27.461493 17963 heartbeater.cc:347] Connected to a master server at 127.8.29.254:41871
I20260923 17:29:27.461844 17963 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:27.462844 17963 heartbeater.cc:511] Master 127.8.29.254:41871 requested a full tablet report, sending...
I20260923 17:29:27.464439 17646 ts_manager.cc:194] Registered new tserver with Master: 8e1df087d7dc40968dcde345e72ec32a (127.8.29.194:44947)
I20260923 17:29:27.465246 17646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.194:55983
W20260923 17:29:27.781301 17966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:27.781608 17966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:27.781714 17966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:27.789042 17966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:27.789170 17966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:27.789295 17966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:29:27.798769 17966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41871
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:27.800096 17966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:27.801973 17966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:27.814949 17971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:27.814920 17974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:27.814998 17972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:27.815459 17966 server_base.cc:1034] running on GCE node
I20260923 17:29:27.816229 17966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:27.817185 17966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:27.818393 17966 hybrid_clock.cc:648] HybridClock initialized: now 1790184567818339 us; error 41 us; skew 500 ppm
I20260923 17:29:27.818733 17966 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:29:27.820916 17966 webserver.cc:460] Webserver started at http://127.8.29.195:34265/ using document root <none> and password file <none>
I20260923 17:29:27.821593 17966 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:27.821712 17966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:27.822013 17966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:27.824826 17966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "e2a056359e7d471eba4d2d3e1273ae42"
format_stamp: "Formatted at 2026-09-23 17:29:27 on dist-test-slave-p7jt"
server_key: "0444ff574b62bd2b41413dda670a1cde"
server_key_iv: "97c61d020a066453853c3fe6c4f9f06e"
server_key_version: "encryptionkey@0"
I20260923 17:29:27.825621 17966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "e2a056359e7d471eba4d2d3e1273ae42"
format_stamp: "Formatted at 2026-09-23 17:29:27 on dist-test-slave-p7jt"
server_key: "0444ff574b62bd2b41413dda670a1cde"
server_key_iv: "97c61d020a066453853c3fe6c4f9f06e"
server_key_version: "encryptionkey@0"
I20260923 17:29:27.830856 17966 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:29:27.834489 17980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:27.835896 17966 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:29:27.836050 17966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "e2a056359e7d471eba4d2d3e1273ae42"
format_stamp: "Formatted at 2026-09-23 17:29:27 on dist-test-slave-p7jt"
server_key: "0444ff574b62bd2b41413dda670a1cde"
server_key_iv: "97c61d020a066453853c3fe6c4f9f06e"
server_key_version: "encryptionkey@0"
I20260923 17:29:27.836251 17966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:27.856719 17966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:27.857633 17966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:27.857932 17966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:27.858963 17966 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:27.860512 17966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:27.860617 17966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:27.860723 17966 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:27.860790 17966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:27.893380 17966 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:34015
I20260923 17:29:27.893445 18092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:34015 every 8 connection(s)
I20260923 17:29:27.894775 17966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:29:27.897141 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 17966
I20260923 17:29:27.897341 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:29:27.897876 8311 external_mini_cluster.cc:1428] Setting key 2e6ed57d614897016b6b17f04d2036f4
I20260923 17:29:27.915650 17834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:27.898523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:38789 (local address 127.8.29.254:41871)
0923 17:29:27.898736 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:27.898742 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:27.899492 (+ 750us) server_negotiation.cc:407] Connection header received
0923 17:29:27.901507 (+ 2015us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:27.901511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:27.901577 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:27.901673 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:27.902730 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.903486 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.905238 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.905626 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.905726 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:27.914858 (+ 9132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:27.914864 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:27.914868 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:27.914887 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:27.914929 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:27.914944 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:27.915026 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:27.915291 (+ 265us) server_negotiation.cc:299] Negotiation successful
0923 17:29:27.915403 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:29:27.916687 18095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:27.898156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:41871 (local address 127.8.29.195:38789)
0923 17:29:27.899328 (+ 1172us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:27.899384 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:27.901274 (+ 1890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:27.901851 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:27.901860 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:27.901952 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:27.902601 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:27.902622 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.903686 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.903691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:27.905111 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:27.905126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.905855 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.905860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:27.906125 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:27.914415 (+ 8290us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:29:27.914419 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:27.914505 (+ 86us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:29:27.914632 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:27.915039 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:27.915046 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:27.915052 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:29:27.915211 (+ 159us) client_negotiation.cc:240] Negotiation successful
0923 17:29:27.915584 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":376,"threads_started":1}
I20260923 17:29:27.918056 18093 heartbeater.cc:347] Connected to a master server at 127.8.29.254:41871
I20260923 17:29:27.918381 18093 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:27.919229 18093 heartbeater.cc:511] Master 127.8.29.254:41871 requested a full tablet report, sending...
I20260923 17:29:27.920792 17646 ts_manager.cc:194] Registered new tserver with Master: e2a056359e7d471eba4d2d3e1273ae42 (127.8.29.195:34015)
I20260923 17:29:27.921597 17646 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.195:38789
I20260923 17:29:27.925542 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:29:27.931411 17834 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:27.929731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57468 (local address 127.8.29.254:41871)
0923 17:29:27.929884 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:27.929889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:27.930377 (+ 488us) server_negotiation.cc:407] Connection header received
0923 17:29:27.930604 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:27.930609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:27.930679 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:27.930762 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:27.931223 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57468: BlockingRecv error: recv got EOF from 127.0.0.1:57468 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:29:27.932092 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17705
I20260923 17:29:27.939172 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:27.932982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58926 (local address 127.8.29.193:38979)
0923 17:29:27.933454 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:27.933462 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:27.933487 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:29:27.933538 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:27.933544 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:27.933764 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:27.933885 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:27.934181 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.935191 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.936358 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:27.936894 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:27.936985 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:27.937842 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:27.937850 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:27.937864 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:27.938449 (+ 585us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:27.938533 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:27.938554 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:27.938669 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:27.938749 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 17:29:27.938877 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":247,"threads_started":1}
I20260923 17:29:27.940505 17827 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:28.021394 17832 heartbeater.cc:503] Master 127.8.29.254:41871 was elected leader, sending a full tablet report...
I20260923 17:29:28.022615 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 17705
W20260923 17:29:28.035661 17630 connection.cc:535] server connection from 127.8.29.193:38983 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 17:29:28.035920 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17836
I20260923 17:29:28.043926 18104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:28.036761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42770 (local address 127.8.29.194:44947)
0923 17:29:28.037310 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:28.037317 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:28.037344 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:29:28.037407 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:28.037414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:28.037625 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:28.037778 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:28.038192 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:28.039233 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:28.040544 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:28.041190 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:28.041307 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:28.042353 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:28.042361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:28.042375 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:28.043103 (+ 728us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:28.043152 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:28.043178 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:28.043280 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:28.043430 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 17:29:28.043588 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":279,"threads_started":1}
I20260923 17:29:28.045221 17958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:28.126566 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 17836
I20260923 17:29:28.142339 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17966
I20260923 17:29:28.150604 18106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:28.143334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38318 (local address 127.8.29.195:34015)
0923 17:29:28.143958 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:28.143965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:28.143996 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:29:28.144056 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:28.144063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:28.144287 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:28.144412 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:28.144852 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:28.145884 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:28.147349 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:28.147951 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:28.148055 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:28.149101 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:28.149110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:28.149125 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:28.149722 (+ 597us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:28.149770 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:28.149790 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:28.149880 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:28.150078 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 17:29:28.150202 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":316,"threads_started":1}
I20260923 17:29:28.151892 18088 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:28.234043 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 17966
I20260923 17:29:28.246718 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 17615
I20260923 17:29:28.247908 17676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:28.351176 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 17615
2026-09-23T17:29:28Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2411 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T17:29:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:28Z Disabled control of system clock
I20260923 17:29:28.418342 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:46075
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:46075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:29:28.753062 18112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:28.753341 18112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:28.753430 18112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:28.760843 18112 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:28.760975 18112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:28.761021 18112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:28.761077 18112 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:28.761121 18112 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:28.770555 18112 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:46075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:46075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:28.771793 18112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:28.773528 18112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:28.784927 18118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:28.784986 18117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:28.785014 18120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:28.785642 18112 server_base.cc:1034] running on GCE node
I20260923 17:29:28.786434 18112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:28.787919 18112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:28.789111 18112 hybrid_clock.cc:648] HybridClock initialized: now 1790184568789069 us; error 41 us; skew 500 ppm
I20260923 17:29:28.789428 18112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:29:28.791574 18112 webserver.cc:460] Webserver started at http://127.8.29.254:44881/ using document root <none> and password file <none>
I20260923 17:29:28.792218 18112 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:28.792302 18112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:28.792577 18112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:28.795465 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "bd1d5712a5004b58b6ab3adff587b0e7"
format_stamp: "Formatted at 2026-09-23 17:29:28 on dist-test-slave-p7jt"
server_key: "94a589da2186ab36acd49707d4b68fb2"
server_key_iv: "921c446283ce13b2c0164be0c4b3eeda"
server_key_version: "encryptionkey@0"
I20260923 17:29:28.796193 18112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "bd1d5712a5004b58b6ab3adff587b0e7"
format_stamp: "Formatted at 2026-09-23 17:29:28 on dist-test-slave-p7jt"
server_key: "94a589da2186ab36acd49707d4b68fb2"
server_key_iv: "921c446283ce13b2c0164be0c4b3eeda"
server_key_version: "encryptionkey@0"
I20260923 17:29:28.801313 18112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:29:28.804865 18126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:28.806305 18112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:29:28.806509 18112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "bd1d5712a5004b58b6ab3adff587b0e7"
format_stamp: "Formatted at 2026-09-23 17:29:28 on dist-test-slave-p7jt"
server_key: "94a589da2186ab36acd49707d4b68fb2"
server_key_iv: "921c446283ce13b2c0164be0c4b3eeda"
server_key_version: "encryptionkey@0"
I20260923 17:29:28.806737 18112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:28.812289 18112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:28.813064 18112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:28.813328 18112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:28.829604 18112 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:46075
I20260923 17:29:28.829622 18177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:46075 every 8 connection(s)
I20260923 17:29:28.831008 18112 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:28.834893 18178 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:28.839025 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 18112
I20260923 17:29:28.839200 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:29:28.839824 8311 external_mini_cluster.cc:1428] Setting key be8fa3f00bac811c86febd2dfe9ca598
I20260923 17:29:28.846045 18178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7: Bootstrap starting.
I20260923 17:29:28.848460 18181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:28.841948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.8.29.254:46075)
0923 17:29:28.843109 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:28.843124 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:28.843173 (+ 49us) server_negotiation.cc:407] Connection header received
0923 17:29:28.844919 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:28.844944 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:28.845385 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:28.845763 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:28.846810 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:28.846816 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:28.846829 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:28.846843 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:28.846897 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:28.846916 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:28.847109 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:28.847247 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:29:28.847519 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":394,"threads_started":1}
I20260923 17:29:28.849642 18178 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:28.850756 18178 log.cc:824] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:28.853467 18178 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7: No bootstrap required, opened a new log
I20260923 17:29:28.860697 18178 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d5712a5004b58b6ab3adff587b0e7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46075 } }
I20260923 17:29:28.861079 18178 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:28.861172 18178 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd1d5712a5004b58b6ab3adff587b0e7, State: Initialized, Role: FOLLOWER
I20260923 17:29:28.861794 18178 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d5712a5004b58b6ab3adff587b0e7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46075 } }
I20260923 17:29:28.862023 18178 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:28.862147 18178 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:28.862304 18178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:28.864760 18178 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d5712a5004b58b6ab3adff587b0e7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46075 } }
I20260923 17:29:28.865267 18178 leader_election.cc:302] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd1d5712a5004b58b6ab3adff587b0e7; no voters:
I20260923 17:29:28.865837 18178 leader_election.cc:288] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:28.865962 18183 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:28.866623 18183 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [term 1 LEADER]: Becoming Leader. State: Replica: bd1d5712a5004b58b6ab3adff587b0e7, State: Running, Role: LEADER
I20260923 17:29:28.867189 18183 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d5712a5004b58b6ab3adff587b0e7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46075 } }
I20260923 17:29:28.867866 18178 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:28.869930 18184 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd1d5712a5004b58b6ab3adff587b0e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d5712a5004b58b6ab3adff587b0e7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46075 } } }
I20260923 17:29:28.870038 18185 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd1d5712a5004b58b6ab3adff587b0e7. Latest consensus state: current_term: 1 leader_uuid: "bd1d5712a5004b58b6ab3adff587b0e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd1d5712a5004b58b6ab3adff587b0e7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46075 } } }
I20260923 17:29:28.870330 18185 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:28.870328 18184 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:28.872804 18190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:28.878386 18190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:28.889019 18190 catalog_manager.cc:1347] Generated new cluster ID: 56f8d9fe6b2a48d59212522173a4f176
I20260923 17:29:28.889150 18190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:28.912438 18190 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:28.912932 18190 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:28.921780 18190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd1d5712a5004b58b6ab3adff587b0e7: Generated new TSK 0
I20260923 17:29:28.923085 18190 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:29:28.940719 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:46075
--builtin_ntp_servers=127.8.29.212:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 17:29:29.274040 18202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:29.274339 18202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:29.274451 18202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:29.281859 18202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:29.281988 18202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:29.282133 18202 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:29:29.291517 18202 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:46075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:29.292874 18202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:29.294708 18202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:29.307873 18210 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:29.307814 18207 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:29.307873 18208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:29.308524 18202 server_base.cc:1034] running on GCE node
I20260923 17:29:29.309094 18202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:29.310030 18202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:29.311270 18202 hybrid_clock.cc:648] HybridClock initialized: now 1790184569311188 us; error 59 us; skew 500 ppm
I20260923 17:29:29.311582 18202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:29:29.313818 18202 webserver.cc:460] Webserver started at http://127.8.29.193:43695/ using document root <none> and password file <none>
I20260923 17:29:29.314534 18202 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:29.314631 18202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:29.314901 18202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:29.317747 18202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "44381e5f69b448f5be530357a9d80581"
format_stamp: "Formatted at 2026-09-23 17:29:29 on dist-test-slave-p7jt"
server_key: "3a4c76753e4acd6eb11cfc097b928c7f"
server_key_iv: "88af9baea2f0c221decd93cbbddc6489"
server_key_version: "encryptionkey@0"
I20260923 17:29:29.318578 18202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "44381e5f69b448f5be530357a9d80581"
format_stamp: "Formatted at 2026-09-23 17:29:29 on dist-test-slave-p7jt"
server_key: "3a4c76753e4acd6eb11cfc097b928c7f"
server_key_iv: "88af9baea2f0c221decd93cbbddc6489"
server_key_version: "encryptionkey@0"
I20260923 17:29:29.324044 18202 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:29:29.327801 18216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:29.329267 18202 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 17:29:29.329429 18202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "44381e5f69b448f5be530357a9d80581"
format_stamp: "Formatted at 2026-09-23 17:29:29 on dist-test-slave-p7jt"
server_key: "3a4c76753e4acd6eb11cfc097b928c7f"
server_key_iv: "88af9baea2f0c221decd93cbbddc6489"
server_key_version: "encryptionkey@0"
I20260923 17:29:29.329648 18202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:29.335649 18202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:29.336526 18202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:29.336763 18202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:29.337749 18202 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:29.339267 18202 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:29.339352 18202 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:29.339439 18202 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:29.339490 18202 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:29.369974 18202 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:33437
I20260923 17:29:29.370023 18328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:33437 every 8 connection(s)
I20260923 17:29:29.371394 18202 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:29:29.375283 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 18202
I20260923 17:29:29.375478 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:29:29.376219 8311 external_mini_cluster.cc:1428] Setting key 10665c5f1460e7449b36d62351b8a655
I20260923 17:29:29.378926 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:46075
--builtin_ntp_servers=127.8.29.212:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 17:29:29.380036 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:29.374066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:55703 (local address 127.8.29.254:46075)
0923 17:29:29.374510 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:29.374517 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:29.375917 (+ 1400us) server_negotiation.cc:407] Connection header received
0923 17:29:29.377673 (+ 1756us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:29.377679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:29.377747 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:29.377838 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:29.379287 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:29.379293 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:29.379298 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:29.379321 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:29.379376 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:29.379389 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:29.379498 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:29.379692 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 17:29:29.379759 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":218,"threads_started":1}
I20260923 17:29:29.381083 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:29.374624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46075 (local address 127.8.29.193:55703)
0923 17:29:29.375753 (+ 1129us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:29.375791 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:29.377391 (+ 1600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:29.378118 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:29.378127 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:29.378233 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:29.378919 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:29:29.378922 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:29.378962 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:29:29.379107 (+ 145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:29.379475 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:29.379486 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:29.379494 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:29:29.379767 (+ 273us) client_negotiation.cc:240] Negotiation successful
0923 17:29:29.380069 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":408,"threads_started":1}
I20260923 17:29:29.382804 18329 heartbeater.cc:347] Connected to a master server at 127.8.29.254:46075
I20260923 17:29:29.383157 18329 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:29.383776 18329 heartbeater.cc:511] Master 127.8.29.254:46075 requested a full tablet report, sending...
I20260923 17:29:29.386075 18143 ts_manager.cc:194] Registered new tserver with Master: 44381e5f69b448f5be530357a9d80581 (127.8.29.193:33437)
W20260923 17:29:29.705309 18333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:29.705588 18333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:29.705673 18333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:29.713028 18333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:29.713140 18333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:29.713265 18333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:29:29.722697 18333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:46075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:29.723969 18333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:29.725808 18333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:29.739603 18338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:29.739588 18341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:29.739610 18339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:29.740289 18333 server_base.cc:1034] running on GCE node
I20260923 17:29:29.740998 18333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:29.742136 18333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:29.743363 18333 hybrid_clock.cc:648] HybridClock initialized: now 1790184569743317 us; error 35 us; skew 500 ppm
I20260923 17:29:29.743707 18333 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:29:29.746121 18333 webserver.cc:460] Webserver started at http://127.8.29.194:46653/ using document root <none> and password file <none>
I20260923 17:29:29.746882 18333 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:29.746989 18333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:29.747316 18333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:29.750248 18333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "73b9288f44134142be8fcdc7c3ad61b2"
format_stamp: "Formatted at 2026-09-23 17:29:29 on dist-test-slave-p7jt"
server_key: "4623c05304c732fea882547c65d7540f"
server_key_iv: "3086d7a1d22460fde3a3babb0895e7cb"
server_key_version: "encryptionkey@0"
I20260923 17:29:29.751139 18333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "73b9288f44134142be8fcdc7c3ad61b2"
format_stamp: "Formatted at 2026-09-23 17:29:29 on dist-test-slave-p7jt"
server_key: "4623c05304c732fea882547c65d7540f"
server_key_iv: "3086d7a1d22460fde3a3babb0895e7cb"
server_key_version: "encryptionkey@0"
I20260923 17:29:29.756711 18333 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:29:29.760627 18347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:29.762260 18333 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:29:29.762568 18333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "73b9288f44134142be8fcdc7c3ad61b2"
format_stamp: "Formatted at 2026-09-23 17:29:29 on dist-test-slave-p7jt"
server_key: "4623c05304c732fea882547c65d7540f"
server_key_iv: "3086d7a1d22460fde3a3babb0895e7cb"
server_key_version: "encryptionkey@0"
I20260923 17:29:29.762852 18333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:29.769030 18333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:29.769976 18333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:29.770268 18333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:29.771325 18333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:29.772997 18333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:29.773100 18333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:29.773209 18333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:29.773279 18333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:29.802362 18333 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:46793
I20260923 17:29:29.802414 18459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:46793 every 8 connection(s)
I20260923 17:29:29.803691 18333 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:29:29.812320 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:29.806167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:45295 (local address 127.8.29.254:46075)
0923 17:29:29.806386 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:29.806392 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:29.807942 (+ 1550us) server_negotiation.cc:407] Connection header received
0923 17:29:29.809693 (+ 1751us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:29.809698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:29.809758 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:29.809856 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:29.811522 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:29.811529 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:29.811533 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:29.811553 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:29.811587 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:29.811601 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:29.811711 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:29.811923 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 17:29:29.812021 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:29:29.812369 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 18333
I20260923 17:29:29.812536 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:29:29.812999 8311 external_mini_cluster.cc:1428] Setting key 6c09ea792eed18d482a87e564ffd7e25
I20260923 17:29:29.813330 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:29.806771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46075 (local address 127.8.29.194:45295)
0923 17:29:29.807783 (+ 1012us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:29.807828 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:29.809368 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:29.810179 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:29.810193 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:29.810367 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:29.811118 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:29:29.811121 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:29.811165 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:29:29.811268 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:29.811699 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:29.811708 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:29.811722 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 17:29:29.811978 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 17:29:29.812253 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":315,"threads_started":1}
I20260923 17:29:29.815039 18460 heartbeater.cc:347] Connected to a master server at 127.8.29.254:46075
I20260923 17:29:29.815347 18460 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:29.815902 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:46075
--builtin_ntp_servers=127.8.29.212:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 17:29:29.815945 18460 heartbeater.cc:511] Master 127.8.29.254:46075 requested a full tablet report, sending...
I20260923 17:29:29.817502 18143 ts_manager.cc:194] Registered new tserver with Master: 73b9288f44134142be8fcdc7c3ad61b2 (127.8.29.194:46793)
W20260923 17:29:30.145534 18463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:30.145828 18463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:30.145917 18463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:30.153225 18463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:30.153342 18463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:30.153450 18463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:29:30.163254 18463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:46075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:30.164522 18463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:30.166340 18463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:30.179678 18468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:30.179746 18469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:30.179708 18471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:30.181089 18463 server_base.cc:1034] running on GCE node
I20260923 17:29:30.181707 18463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:30.182698 18463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:30.183892 18463 hybrid_clock.cc:648] HybridClock initialized: now 1790184570183832 us; error 56 us; skew 500 ppm
I20260923 17:29:30.184206 18463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:29:30.186542 18463 webserver.cc:460] Webserver started at http://127.8.29.195:34633/ using document root <none> and password file <none>
I20260923 17:29:30.187239 18463 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:30.187330 18463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:30.187610 18463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:30.190605 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "bf495825153448d5a3f304ae935e34aa"
format_stamp: "Formatted at 2026-09-23 17:29:30 on dist-test-slave-p7jt"
server_key: "80f40e1600ed160d4c73fdb2d1e2dce1"
server_key_iv: "3c68f80a2b54c3b1b274e671f19aff3f"
server_key_version: "encryptionkey@0"
I20260923 17:29:30.191422 18463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "bf495825153448d5a3f304ae935e34aa"
format_stamp: "Formatted at 2026-09-23 17:29:30 on dist-test-slave-p7jt"
server_key: "80f40e1600ed160d4c73fdb2d1e2dce1"
server_key_iv: "3c68f80a2b54c3b1b274e671f19aff3f"
server_key_version: "encryptionkey@0"
I20260923 17:29:30.196943 18463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:29:30.200555 18477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:30.202121 18463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:29:30.202327 18463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "bf495825153448d5a3f304ae935e34aa"
format_stamp: "Formatted at 2026-09-23 17:29:30 on dist-test-slave-p7jt"
server_key: "80f40e1600ed160d4c73fdb2d1e2dce1"
server_key_iv: "3c68f80a2b54c3b1b274e671f19aff3f"
server_key_version: "encryptionkey@0"
I20260923 17:29:30.202589 18463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:30.208637 18463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:30.209475 18463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:30.209715 18463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:30.210738 18463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:30.212224 18463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:30.212308 18463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:30.212393 18463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:30.212443 18463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:30.244367 18463 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:38105
I20260923 17:29:30.244405 18589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:38105 every 8 connection(s)
I20260923 17:29:30.245769 18463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:29:30.248508 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 18463
I20260923 17:29:30.248705 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:29:30.249229 8311 external_mini_cluster.cc:1428] Setting key aade243c2ac73c276659d798fbc8f6cb
I20260923 17:29:30.254833 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:30.248468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:33947 (local address 127.8.29.254:46075)
0923 17:29:30.248652 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:30.248663 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:30.250612 (+ 1949us) server_negotiation.cc:407] Connection header received
0923 17:29:30.252571 (+ 1959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:30.252576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:30.252642 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:30.252740 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:30.254083 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:30.254089 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:30.254094 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:30.254113 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:30.254145 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:30.254156 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:30.254250 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:30.254493 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 17:29:30.254560 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:29:30.255751 18592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:30.249215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46075 (local address 127.8.29.195:33947)
0923 17:29:30.250413 (+ 1198us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:30.250497 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:30.252312 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:30.252999 (+ 687us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:30.253008 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:30.253110 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:30.253735 (+ 625us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:29:30.253738 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:30.253782 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:29:30.253874 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:30.254240 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:30.254250 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:30.254259 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:29:30.254539 (+ 280us) client_negotiation.cc:240] Negotiation successful
0923 17:29:30.254810 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":334,"threads_started":1}
I20260923 17:29:30.257097 18590 heartbeater.cc:347] Connected to a master server at 127.8.29.254:46075
I20260923 17:29:30.257388 18590 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:30.257987 18590 heartbeater.cc:511] Master 127.8.29.254:46075 requested a full tablet report, sending...
I20260923 17:29:30.259425 18143 ts_manager.cc:194] Registered new tserver with Master: bf495825153448d5a3f304ae935e34aa (127.8.29.195:38105)
I20260923 17:29:30.264732 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 17:29:30.270974 18331 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:29:30.269295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40704 (local address 127.8.29.254:46075)
0923 17:29:30.269456 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:30.269460 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:30.270027 (+ 567us) server_negotiation.cc:407] Connection header received
0923 17:29:30.270187 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:30.270193 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:30.270255 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:30.270337 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:30.270721 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40704: BlockingRecv error: recv got EOF from 127.0.0.1:40704 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:29:30.271751 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 18202
I20260923 17:29:30.275192 18599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:30.272681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.8.29.193:33437)
0923 17:29:30.273153 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:30.273161 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:30.273187 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:29:30.273251 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:30.273258 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:30.273504 (+ 246us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:30.273633 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:30.274610 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:30.274618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:30.274631 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:30.274643 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:30.274691 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:30.274709 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:30.274813 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:30.274885 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 17:29:30.274970 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":206,"threads_started":1}
I20260923 17:29:30.276378 18324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:30.355693 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 18202
I20260923 17:29:30.370126 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 18333
I20260923 17:29:30.373898 18601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:30.371072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42732 (local address 127.8.29.194:46793)
0923 17:29:30.371564 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:30.371571 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:30.371597 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:29:30.371661 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:30.371667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:30.371889 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:30.372032 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:30.373127 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:30.373134 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:30.373157 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:30.373175 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:30.373220 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:30.373236 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:30.373363 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:30.373501 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:29:30.373609 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
I20260923 17:29:30.375356 18455 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:30.455888 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 18333
I20260923 17:29:30.468852 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 18463
I20260923 17:29:30.472909 18603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:30.469824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43818 (local address 127.8.29.195:38105)
0923 17:29:30.470438 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:30.470449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:30.470509 (+ 60us) server_negotiation.cc:407] Connection header received
0923 17:29:30.470579 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:30.470585 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:30.470823 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:30.471018 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:30.472142 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:30.472148 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:30.472161 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:29:30.472178 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:30.472221 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:29:30.472234 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:30.472387 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:30.472463 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 17:29:30.472565 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":310,"threads_started":1}
I20260923 17:29:30.474236 18585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:30.559104 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 18463
I20260923 17:29:30.572602 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 18112
I20260923 17:29:30.573828 18173 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:30.674330 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 18112
2026-09-23T17:29:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2322 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:30 dist-test-slave-p7jt krb5kdc[18608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:30 dist-test-slave-p7jt krb5kdc[18608](info): set up 2 sockets
Sep 23 17:29:30 dist-test-slave-p7jt krb5kdc[18608](info): commencing operation
krb5kdc: starting...
W20260923 17:29:31.715201 8311 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.003s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:29:31 dist-test-slave-p7jt krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:29:31.945986 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:35075
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:35075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:32.279433 18621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:32.279767 18621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:32.279893 18621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:32.287338 18621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:29:32.287465 18621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:32.287539 18621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:32.287613 18621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:29:32.287673 18621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:29:32.297924 18621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:35075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:35075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:32.299342 18621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:32.300693 18621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:32.312532 18627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:32.312558 18626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:32.312498 18629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:32.313292 18621 server_base.cc:1034] running on GCE node
I20260923 17:29:32.314107 18621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:32.315652 18621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:32.316845 18621 hybrid_clock.cc:648] HybridClock initialized: now 1790184572316789 us; error 52 us; skew 500 ppm
I20260923 17:29:32.317180 18621 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:32 dist-test-slave-p7jt krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184572, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:32.320291 18621 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:29:32.321779 18621 webserver.cc:460] Webserver started at http://127.8.29.254:42973/ using document root <none> and password file <none>
I20260923 17:29:32.322512 18621 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:32.322630 18621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:32.322924 18621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:32.325924 18621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "5150ea7d8ae44d92909f5aba4ceb219f"
format_stamp: "Formatted at 2026-09-23 17:29:32 on dist-test-slave-p7jt"
server_key: "3e6f9ae1eb3ef4fc4484ab7ca33d2654"
server_key_iv: "da0349ccf1d4edf711ce395982299328"
server_key_version: "encryptionkey@0"
I20260923 17:29:32.326746 18621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "5150ea7d8ae44d92909f5aba4ceb219f"
format_stamp: "Formatted at 2026-09-23 17:29:32 on dist-test-slave-p7jt"
server_key: "3e6f9ae1eb3ef4fc4484ab7ca33d2654"
server_key_iv: "da0349ccf1d4edf711ce395982299328"
server_key_version: "encryptionkey@0"
I20260923 17:29:32.331914 18621 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:29:32.335359 18636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:32.336704 18621 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:29:32.336872 18621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "5150ea7d8ae44d92909f5aba4ceb219f"
format_stamp: "Formatted at 2026-09-23 17:29:32 on dist-test-slave-p7jt"
server_key: "3e6f9ae1eb3ef4fc4484ab7ca33d2654"
server_key_iv: "da0349ccf1d4edf711ce395982299328"
server_key_version: "encryptionkey@0"
I20260923 17:29:32.337061 18621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:32.349174 18621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:32.353353 18621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:32.353636 18621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:32.371248 18621 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:35075
I20260923 17:29:32.371270 18687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:35075 every 8 connection(s)
I20260923 17:29:32.372644 18621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:29:32.376446 18688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:32.378190 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 18621
I20260923 17:29:32.378386 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:29:32.379215 8311 external_mini_cluster.cc:1428] Setting key 1445b0cbc114ded66eae815689170c7e
I20260923 17:29:32.388389 18688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f: Bootstrap starting.
I20260923 17:29:32.391659 18688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:32.392762 18688 log.cc:824] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:29:32 dist-test-slave-p7jt krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:29:32.396130 18688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f: No bootstrap required, opened a new log
I20260923 17:29:32.401921 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:32.381605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49976 (local address 127.8.29.254:35075)
0923 17:29:32.382404 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:32.382417 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:32.382483 (+ 66us) server_negotiation.cc:407] Connection header received
0923 17:29:32.384159 (+ 1676us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:32.384184 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:32.384615 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:32.384992 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:32.386513 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:32.387810 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:32.389069 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:32.389781 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:32.389962 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:32.394385 (+ 4423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:32.394398 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:32.394422 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:32.394499 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:32.397615 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:32.398352 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:32.398359 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:32.398366 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:32.398608 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:32.398991 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:32.398996 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:32.399000 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:32.399581 (+ 581us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:32.399866 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:32.400344 (+ 478us) server_negotiation.cc:299] Negotiation successful
0923 17:29:32.400822 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":329,"threads_started":1}
I20260923 17:29:32.404294 18688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5150ea7d8ae44d92909f5aba4ceb219f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35075 } }
I20260923 17:29:32.404755 18688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:32.404875 18688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5150ea7d8ae44d92909f5aba4ceb219f, State: Initialized, Role: FOLLOWER
I20260923 17:29:32.405455 18688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5150ea7d8ae44d92909f5aba4ceb219f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35075 } }
I20260923 17:29:32.405663 18688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:29:32.405755 18688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:29:32.405876 18688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:32.408027 18688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5150ea7d8ae44d92909f5aba4ceb219f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35075 } }
I20260923 17:29:32.408470 18688 leader_election.cc:302] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5150ea7d8ae44d92909f5aba4ceb219f; no voters:
I20260923 17:29:32.408972 18688 leader_election.cc:288] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:29:32.409101 18693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:32.410259 18693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [term 1 LEADER]: Becoming Leader. State: Replica: 5150ea7d8ae44d92909f5aba4ceb219f, State: Running, Role: LEADER
I20260923 17:29:32.410871 18693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5150ea7d8ae44d92909f5aba4ceb219f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35075 } }
I20260923 17:29:32.411161 18688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:29:32.413496 18694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5150ea7d8ae44d92909f5aba4ceb219f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5150ea7d8ae44d92909f5aba4ceb219f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35075 } } }
I20260923 17:29:32.413908 18694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:32.414558 18695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5150ea7d8ae44d92909f5aba4ceb219f. Latest consensus state: current_term: 1 leader_uuid: "5150ea7d8ae44d92909f5aba4ceb219f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5150ea7d8ae44d92909f5aba4ceb219f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 35075 } } }
I20260923 17:29:32.414889 18695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f [sys.catalog]: This master's current role is: LEADER
I20260923 17:29:32.416795 18701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:29:32.421902 18701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:29:32.430989 18701 catalog_manager.cc:1347] Generated new cluster ID: cdd9fc3587444ad39529c40d262f8758
I20260923 17:29:32.431116 18701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:29:32.442162 18701 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:29:32.444224 18701 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:29:32.453063 18701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f: Generated new TSK 0
I20260923 17:29:32.453997 18701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:29:32.509871 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:35075
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:32.858006 18714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:32.858300 18714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:32.858394 18714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:32.865739 18714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:32.865862 18714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:32.865981 18714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:29:32.875933 18714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:35075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:32.877274 18714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:32.878541 18714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:32.891372 18722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:32.891413 18719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:32.891438 18720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:32.892213 18714 server_base.cc:1034] running on GCE node
I20260923 17:29:32.892840 18714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:32.893775 18714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:32.894965 18714 hybrid_clock.cc:648] HybridClock initialized: now 1790184572894926 us; error 33 us; skew 500 ppm
I20260923 17:29:32.895303 18714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:32 dist-test-slave-p7jt krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184572, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:32.898320 18714 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:29:32.899740 18714 webserver.cc:460] Webserver started at http://127.8.29.193:39539/ using document root <none> and password file <none>
I20260923 17:29:32.900420 18714 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:32.900537 18714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:32.900892 18714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:32.904435 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "515f3f9c129d4782be2551d43f1c783c"
format_stamp: "Formatted at 2026-09-23 17:29:32 on dist-test-slave-p7jt"
server_key: "a04872d32001bf9d243608a151c70485"
server_key_iv: "4e52ea7c8ca8580a968e4573e44e084a"
server_key_version: "encryptionkey@0"
I20260923 17:29:32.905479 18714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "515f3f9c129d4782be2551d43f1c783c"
format_stamp: "Formatted at 2026-09-23 17:29:32 on dist-test-slave-p7jt"
server_key: "a04872d32001bf9d243608a151c70485"
server_key_iv: "4e52ea7c8ca8580a968e4573e44e084a"
server_key_version: "encryptionkey@0"
I20260923 17:29:32.911082 18714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260923 17:29:32.914873 18729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:32.916255 18714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:29:32.916419 18714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "515f3f9c129d4782be2551d43f1c783c"
format_stamp: "Formatted at 2026-09-23 17:29:32 on dist-test-slave-p7jt"
server_key: "a04872d32001bf9d243608a151c70485"
server_key_iv: "4e52ea7c8ca8580a968e4573e44e084a"
server_key_version: "encryptionkey@0"
I20260923 17:29:32.916618 18714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:32.935802 18714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:32.940052 18714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:32.940351 18714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:32.941395 18714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:32.942783 18714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:32.942854 18714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:32.942941 18714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:32.942983 18714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:32.973130 18714 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:35275
I20260923 17:29:32.973165 18841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:35275 every 8 connection(s)
I20260923 17:29:32.974397 18714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:29:32.978662 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 18714
I20260923 17:29:32.978865 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:29:32.979499 8311 external_mini_cluster.cc:1428] Setting key 8a6258f90a2b95b70e1c228b7bed2eaf
Sep 23 17:29:32 dist-test-slave-p7jt krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184572, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:29:32.994899 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:32.977122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:56051 (local address 127.8.29.254:35075)
0923 17:29:32.977648 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:32.977653 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:32.978674 (+ 1021us) server_negotiation.cc:407] Connection header received
0923 17:29:32.980949 (+ 2275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:32.980955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:32.981063 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:32.981152 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:32.982801 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:32.983517 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:32.985092 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:32.985488 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:32.985607 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:32.989279 (+ 3672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:32.989285 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:32.989288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:32.989323 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:32.991792 (+ 2469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:32.992535 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:32.992540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:32.992542 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:32.992671 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:32.993136 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:32.993140 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:32.993142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:32.993406 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:32.993538 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:32.994282 (+ 744us) server_negotiation.cc:299] Negotiation successful
0923 17:29:32.994542 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":250,"threads_started":1}
I20260923 17:29:32.995913 18845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:32.977712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35075 (local address 127.8.29.193:56051)
0923 17:29:32.978503 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:32.978560 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:32.980660 (+ 2100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:32.981427 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:32.981436 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:32.981960 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:32.982628 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:32.982650 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:32.983667 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:32.983671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:32.984930 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:32.984942 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:32.985723 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:32.985730 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:32.985974 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:32.986646 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:32.986671 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:32.989024 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:32.991943 (+ 2919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:32.991953 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:32.991973 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:32.992402 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:32.992775 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:32.992780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:32.992784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:32.993019 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:32.993539 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:32.993549 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:32.993900 (+ 351us) client_negotiation.cc:769] Sending connection context
0923 17:29:32.994139 (+ 239us) client_negotiation.cc:240] Negotiation successful
0923 17:29:32.994733 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":313,"threads_started":1}
I20260923 17:29:32.997584 18842 heartbeater.cc:347] Connected to a master server at 127.8.29.254:35075
I20260923 17:29:32.997928 18842 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:32.998905 18842 heartbeater.cc:511] Master 127.8.29.254:35075 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:29:33.001117 18653 ts_manager.cc:194] Registered new tserver with Master: 515f3f9c129d4782be2551d43f1c783c (127.8.29.193:35275)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
I20260923 17:29:33.002772 18653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:56051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:29:33.029434 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:35075
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:33.350860 18848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:33.351158 18848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:33.351254 18848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:33.358628 18848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:33.358747 18848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:33.358857 18848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:29:33.368709 18848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:35075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:33.370101 18848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:33.371562 18848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:33.384476 18853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:33.384476 18854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:33.385156 18856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:33.385422 18848 server_base.cc:1034] running on GCE node
I20260923 17:29:33.386029 18848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:33.387025 18848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:33.388217 18848 hybrid_clock.cc:648] HybridClock initialized: now 1790184573388167 us; error 44 us; skew 500 ppm
I20260923 17:29:33.388523 18848 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:33 dist-test-slave-p7jt krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184573, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:33.391697 18848 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:29:33.393178 18848 webserver.cc:460] Webserver started at http://127.8.29.194:36865/ using document root <none> and password file <none>
I20260923 17:29:33.393890 18848 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:33.393997 18848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:33.394320 18848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:33.397321 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "f76bb6b0f65f47959846462bea6021a5"
format_stamp: "Formatted at 2026-09-23 17:29:33 on dist-test-slave-p7jt"
server_key: "7094da10601fcdb6954bdcefca8648c9"
server_key_iv: "37361ecc1fb24f139a9d1e1f5530ffc8"
server_key_version: "encryptionkey@0"
I20260923 17:29:33.398108 18848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "f76bb6b0f65f47959846462bea6021a5"
format_stamp: "Formatted at 2026-09-23 17:29:33 on dist-test-slave-p7jt"
server_key: "7094da10601fcdb6954bdcefca8648c9"
server_key_iv: "37361ecc1fb24f139a9d1e1f5530ffc8"
server_key_version: "encryptionkey@0"
I20260923 17:29:33.403512 18848 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 17:29:33.407402 18863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:33.408758 18848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:29:33.408916 18848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "f76bb6b0f65f47959846462bea6021a5"
format_stamp: "Formatted at 2026-09-23 17:29:33 on dist-test-slave-p7jt"
server_key: "7094da10601fcdb6954bdcefca8648c9"
server_key_iv: "37361ecc1fb24f139a9d1e1f5530ffc8"
server_key_version: "encryptionkey@0"
I20260923 17:29:33.409112 18848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:33.434345 18848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:33.438711 18848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:33.438987 18848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:33.440028 18848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:33.441448 18848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:33.441521 18848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:33.441632 18848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:33.441679 18848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:33.473245 18848 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:36441
I20260923 17:29:33.473280 18975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:36441 every 8 connection(s)
I20260923 17:29:33.474530 18848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:29:33.481287 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 18848
I20260923 17:29:33.481467 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:29:33.481890 8311 external_mini_cluster.cc:1428] Setting key 5abef03a4a35e79cbf61f6c5e0ac62e3
Sep 23 17:29:33 dist-test-slave-p7jt krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184573, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:29:33.495518 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:33.477049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:48463 (local address 127.8.29.254:35075)
0923 17:29:33.477239 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:33.477244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:33.478476 (+ 1232us) server_negotiation.cc:407] Connection header received
0923 17:29:33.480330 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:33.480335 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:33.480411 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:33.480500 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:33.482205 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:33.483136 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:33.485080 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:33.485735 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:33.485875 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:33.489482 (+ 3607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:33.489489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:33.489493 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:33.489532 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:33.492254 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:33.493144 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:33.493149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:33.493151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:33.493278 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:33.493754 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:33.493758 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:33.493760 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:33.494051 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:33.494199 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:33.495100 (+ 901us) server_negotiation.cc:299] Negotiation successful
0923 17:29:33.495283 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:29:33.496898 18978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:33.477560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35075 (local address 127.8.29.194:48463)
0923 17:29:33.478280 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:33.478330 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:33.480032 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:33.480730 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:33.480740 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:33.481304 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:33.481986 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:33.482011 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:33.483311 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:33.483317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:33.484894 (+ 1577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:33.484907 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:33.485948 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:33.485953 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:33.486164 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:33.486895 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:33.486927 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:33.489233 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:33.492402 (+ 3169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:33.492411 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:33.492431 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:33.493014 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:33.493375 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:33.493380 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:33.493384 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:33.493624 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:33.494215 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:33.494224 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:33.494698 (+ 474us) client_negotiation.cc:769] Sending connection context
0923 17:29:33.494995 (+ 297us) client_negotiation.cc:240] Negotiation successful
0923 17:29:33.495548 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":271,"threads_started":1}
I20260923 17:29:33.498371 18976 heartbeater.cc:347] Connected to a master server at 127.8.29.254:35075
I20260923 17:29:33.498687 18976 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:33.499423 18976 heartbeater.cc:511] Master 127.8.29.254:35075 requested a full tablet report, sending...
I20260923 17:29:33.500927 18653 ts_manager.cc:194] Registered new tserver with Master: f76bb6b0f65f47959846462bea6021a5 (127.8.29.194:36441)
I20260923 17:29:33.502015 18653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:48463
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:29:33.534533 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:35075
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:29:33.854427 18981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:29:33.854795 18981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:29:33.854935 18981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:29:33.862280 18981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:29:33.862416 18981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:29:33.862591 18981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:29:33.872546 18981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:35075
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:29:33.873906 18981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:29:33.875353 18981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:29:33.888154 18986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:33.888127 18987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:29:33.888358 18989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:29:33.888813 18981 server_base.cc:1034] running on GCE node
I20260923 17:29:33.889465 18981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:29:33.890388 18981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:29:33.891619 18981 hybrid_clock.cc:648] HybridClock initialized: now 1790184573891559 us; error 49 us; skew 500 ppm
I20260923 17:29:33.891939 18981 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:29:33 dist-test-slave-p7jt krb5kdc[18608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184573, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:29:33.895147 18981 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:29:33.896575 18981 webserver.cc:460] Webserver started at http://127.8.29.195:44437/ using document root <none> and password file <none>
I20260923 17:29:33.897243 18981 fs_manager.cc:359] Metadata directory not provided
I20260923 17:29:33.897344 18981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:29:33.897595 18981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:29:33.900661 18981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "09001058d9d74a96a94ad5b4daf09594"
format_stamp: "Formatted at 2026-09-23 17:29:33 on dist-test-slave-p7jt"
server_key: "767a43d243170984da4c524765a8b0e6"
server_key_iv: "a80b631ebd69cc1c88db16e355c4293e"
server_key_version: "encryptionkey@0"
I20260923 17:29:33.901401 18981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "09001058d9d74a96a94ad5b4daf09594"
format_stamp: "Formatted at 2026-09-23 17:29:33 on dist-test-slave-p7jt"
server_key: "767a43d243170984da4c524765a8b0e6"
server_key_iv: "a80b631ebd69cc1c88db16e355c4293e"
server_key_version: "encryptionkey@0"
I20260923 17:29:33.906745 18981 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:29:33.911434 18996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:33.912938 18981 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:29:33.913098 18981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "09001058d9d74a96a94ad5b4daf09594"
format_stamp: "Formatted at 2026-09-23 17:29:33 on dist-test-slave-p7jt"
server_key: "767a43d243170984da4c524765a8b0e6"
server_key_iv: "a80b631ebd69cc1c88db16e355c4293e"
server_key_version: "encryptionkey@0"
I20260923 17:29:33.913290 18981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:29:33.943948 18981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:29:33.948088 18981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:29:33.948341 18981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:29:33.949276 18981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:29:33.950668 18981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:29:33.950739 18981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:33.950829 18981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:29:33.950876 18981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:29:33.981721 18981 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:37175
I20260923 17:29:33.981786 19108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:37175 every 8 connection(s)
I20260923 17:29:33.983050 18981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:29:33.986690 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 18981
I20260923 17:29:33.986860 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:29:33.987329 8311 external_mini_cluster.cc:1428] Setting key 5c5069f8693d23aef066786d4f829acc
Sep 23 17:29:33 dist-test-slave-p7jt krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184573, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:29:34.004366 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:33.985596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:42311 (local address 127.8.29.254:35075)
0923 17:29:33.985777 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:33.985783 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:33.987212 (+ 1429us) server_negotiation.cc:407] Connection header received
0923 17:29:33.989076 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:33.989081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:33.989183 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:33.989262 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:33.991092 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:33.991865 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:33.993574 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:33.994006 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:33.994167 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:33.997769 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:33.997775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:33.997779 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:33.997819 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:34.000567 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.001580 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.001585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.001588 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.001775 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.002414 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.002419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.002423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.002788 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:34.002976 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.003900 (+ 924us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.004110 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:29:34.005326 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:33.986149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:35075 (local address 127.8.29.195:42311)
0923 17:29:33.986998 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:33.987065 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:33.988795 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:33.989540 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:33.989551 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:33.990192 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:29:33.990882 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:33.990903 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:33.992022 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:33.992026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:33.993405 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:33.993419 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:33.994251 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:33.994256 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:33.994515 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:33.995139 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:29:33.995162 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:29:33.997475 (+ 2313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:29:34.000771 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:34.000782 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:34.000804 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:34.001403 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:34.001920 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:29:34.001924 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:29:34.001926 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:29:34.002273 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:29:34.002949 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:29:34.002956 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:29:34.003441 (+ 485us) client_negotiation.cc:769] Sending connection context
0923 17:29:34.003733 (+ 292us) client_negotiation.cc:240] Negotiation successful
0923 17:29:34.004189 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":371,"threads_started":1}
I20260923 17:29:34.005756 18842 heartbeater.cc:503] Master 127.8.29.254:35075 was elected leader, sending a full tablet report...
I20260923 17:29:34.006902 19109 heartbeater.cc:347] Connected to a master server at 127.8.29.254:35075
I20260923 17:29:34.007261 19109 heartbeater.cc:464] Registering TS with master...
I20260923 17:29:34.008023 19109 heartbeater.cc:511] Master 127.8.29.254:35075 requested a full tablet report, sending...
I20260923 17:29:34.009402 18653 ts_manager.cc:194] Registered new tserver with Master: 09001058d9d74a96a94ad5b4daf09594 (127.8.29.195:37175)
I20260923 17:29:34.010208 18653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:42311
I20260923 17:29:34.015793 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:29:34.032055 18844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.020383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49990 (local address 127.8.29.254:35075)
0923 17:29:34.020541 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.020546 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.021062 (+ 516us) server_negotiation.cc:407] Connection header received
0923 17:29:34.021176 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.021180 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.021268 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.021366 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:34.022820 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.023573 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.024806 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.025208 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.025314 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.027548 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:34.027553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:34.027557 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:34.027593 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:34.029875 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.030446 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.030450 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.030452 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.030605 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.030909 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.030912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.030914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.031232 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:34.031411 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.031657 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.031821 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:29:34.036160 18653 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:29:34.043293 18653 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:29:34.081460 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.058752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41420 (local address 127.8.29.194:36441)
0923 17:29:34.073672 (+ 14920us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.073681 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.074551 (+ 870us) server_negotiation.cc:407] Connection header received
0923 17:29:34.074749 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.074757 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.075042 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.075240 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:34.075869 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.077117 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.079368 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.080334 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.080473 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.080699 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.080990 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.081172 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14744,"thread_start_us":14671,"threads_started":1}
I20260923 17:29:34.081610 19125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.058201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:36441 (local address 127.0.0.1:41420)
0923 17:29:34.074261 (+ 16060us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:34.074282 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:34.074408 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:34.075311 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:34.075316 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:34.075346 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:34.075718 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.075729 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.077227 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.077232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.079213 (+ 1981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.079228 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.080560 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.080566 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.080763 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.080806 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 17:29:34.080924 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:29:34.081142 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15772,"mutex_wait_us":52,"thread_start_us":15631,"threads_started":1}
I20260923 17:29:34.082293 19122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.056238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:35275 (local address 127.0.0.1:56906)
0923 17:29:34.072227 (+ 15989us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:34.072257 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:34.072436 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:34.074175 (+ 1739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:34.074183 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:34.074231 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:34.074733 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.074746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.076662 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.076667 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.079232 (+ 2565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.079246 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.081466 (+ 2220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.081473 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.081661 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.081696 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:29:34.081772 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 17:29:34.081906 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15799,"mutex_wait_us":42,"thread_start_us":16798,"threads_started":1}
I20260923 17:29:34.083180 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.056376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56906 (local address 127.8.29.193:35275)
0923 17:29:34.073094 (+ 16718us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.073104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.073140 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:29:34.073249 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.073259 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.073583 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.073831 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:34.074942 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.076504 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.079424 (+ 2920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.080916 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.081071 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.081317 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.082656 (+ 1339us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.082843 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16538,"thread_start_us":16388,"threads_started":1}
I20260923 17:29:34.083937 19121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.056174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:37175 (local address 127.0.0.1:36188)
0923 17:29:34.069326 (+ 13152us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:34.069382 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:34.069605 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:34.076446 (+ 6841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:34.076450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:34.076475 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:34.076896 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.076909 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.078865 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.078870 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.080989 (+ 2119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.081003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.083202 (+ 2199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.083209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.083372 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.083403 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:29:34.083487 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:29:34.083629 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12857,"thread_start_us":12671,"threads_started":1}
I20260923 17:29:34.084659 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.056553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36188 (local address 127.8.29.195:37175)
0923 17:29:34.075876 (+ 19323us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.075885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.075919 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:29:34.076017 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.076025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.076247 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.076425 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:34.077098 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.078699 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.081170 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.082741 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.082855 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.083070 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.084156 (+ 1086us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.084318 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19159,"thread_start_us":18972,"threads_started":1}
I20260923 17:29:34.086149 18911 tablet_service.cc:1467] Processing CreateTablet for tablet 1caeeede503b45a1a3381c2227362320 (DEFAULT_TABLE table=test-table [id=d37b2de565bc4d3e961d04a2bbe7a655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:34.087812 18911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1caeeede503b45a1a3381c2227362320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:34.087450 18777 tablet_service.cc:1467] Processing CreateTablet for tablet 1caeeede503b45a1a3381c2227362320 (DEFAULT_TABLE table=test-table [id=d37b2de565bc4d3e961d04a2bbe7a655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:34.087781 19044 tablet_service.cc:1467] Processing CreateTablet for tablet 1caeeede503b45a1a3381c2227362320 (DEFAULT_TABLE table=test-table [id=d37b2de565bc4d3e961d04a2bbe7a655]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:29:34.089063 19044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1caeeede503b45a1a3381c2227362320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:34.089646 18777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1caeeede503b45a1a3381c2227362320. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:29:34.100108 19127 tablet_bootstrap.cc:492] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: Bootstrap starting.
I20260923 17:29:34.103318 19128 tablet_bootstrap.cc:492] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: Bootstrap starting.
I20260923 17:29:34.104139 19127 tablet_bootstrap.cc:654] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:34.105461 19127 log.cc:824] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:34.106264 19129 tablet_bootstrap.cc:492] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: Bootstrap starting.
I20260923 17:29:34.106973 19128 tablet_bootstrap.cc:654] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:34.108348 19128 log.cc:824] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:34.109299 19129 tablet_bootstrap.cc:654] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: Neither blocks nor log segments found. Creating new log.
I20260923 17:29:34.110435 19129 log.cc:824] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:29:34.111120 19127 tablet_bootstrap.cc:492] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: No bootstrap required, opened a new log
I20260923 17:29:34.111416 19127 ts_tablet_manager.cc:1397] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20260923 17:29:34.113107 19128 tablet_bootstrap.cc:492] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: No bootstrap required, opened a new log
I20260923 17:29:34.113444 19128 ts_tablet_manager.cc:1397] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 17:29:34.113869 19129 tablet_bootstrap.cc:492] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: No bootstrap required, opened a new log
I20260923 17:29:34.114161 19129 ts_tablet_manager.cc:1397] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 17:29:34.122117 19129 raft_consensus.cc:348] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } }
I20260923 17:29:34.122113 19127 raft_consensus.cc:348] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } }
I20260923 17:29:34.122629 19127 raft_consensus.cc:374] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:34.122766 19127 raft_consensus.cc:729] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09001058d9d74a96a94ad5b4daf09594, State: Initialized, Role: FOLLOWER
I20260923 17:29:34.122818 19129 raft_consensus.cc:374] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:34.122962 19129 raft_consensus.cc:729] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 515f3f9c129d4782be2551d43f1c783c, State: Initialized, Role: FOLLOWER
I20260923 17:29:34.123602 19129 consensus_queue.cc:260] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } }
I20260923 17:29:34.123628 19127 consensus_queue.cc:260] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } }
I20260923 17:29:34.123592 19128 raft_consensus.cc:348] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } }
I20260923 17:29:34.124081 19128 raft_consensus.cc:374] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:29:34.124255 19128 raft_consensus.cc:729] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f76bb6b0f65f47959846462bea6021a5, State: Initialized, Role: FOLLOWER
I20260923 17:29:34.124943 19128 consensus_queue.cc:260] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } }
I20260923 17:29:34.125427 19129 ts_tablet_manager.cc:1428] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20260923 17:29:34.127188 19109 heartbeater.cc:503] Master 127.8.29.254:35075 was elected leader, sending a full tablet report...
I20260923 17:29:34.128026 19127 ts_tablet_manager.cc:1428] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: Time spent starting tablet: real 0.016s user 0.012s sys 0.003s
I20260923 17:29:34.129347 18976 heartbeater.cc:503] Master 127.8.29.254:35075 was elected leader, sending a full tablet report...
I20260923 17:29:34.130085 19128 ts_tablet_manager.cc:1428] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: Time spent starting tablet: real 0.017s user 0.012s sys 0.004s
I20260923 17:29:34.223807 19134 raft_consensus.cc:482] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:29:34.224149 19134 raft_consensus.cc:504] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } }
I20260923 17:29:34.225881 19134 leader_election.cc:288] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 515f3f9c129d4782be2551d43f1c783c (127.8.29.193:35275), f76bb6b0f65f47959846462bea6021a5 (127.8.29.194:36441)
W20260923 17:29:34.226445 18977 tablet.cc:2369] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:29:34.226797 18843 tablet.cc:2369] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:29:34.233135 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.226154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:36441 (local address 127.8.29.195:39417)
0923 17:29:34.227512 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:34.227532 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:34.227662 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:34.228333 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:34.228339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:34.228369 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:34.228557 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.228570 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.229664 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.229670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.231144 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.231156 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.232393 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.232400 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.232606 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.232643 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 17:29:34.232752 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 17:29:34.232891 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1187}
I20260923 17:29:34.233289 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.227815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:39417 (local address 127.8.29.194:36441)
0923 17:29:34.228025 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.228031 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.228052 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:29:34.228136 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.228141 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.228211 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.228294 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:34.228713 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.229523 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.231312 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.232225 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.232356 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.232410 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.232819 (+ 409us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.232967 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:29:34.233615 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.227149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:34415 (local address 127.8.29.193:35275)
0923 17:29:34.227351 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.227357 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.228155 (+ 798us) server_negotiation.cc:407] Connection header received
0923 17:29:34.228330 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.228336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.228408 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.228523 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:29:34.229085 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.229857 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.231684 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.232565 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.232740 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.232808 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.233167 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.233302 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:29:34.233784 19136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.226447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:35275 (local address 127.8.29.195:34415)
0923 17:29:34.228048 (+ 1601us) negotiation.cc:107] Waiting for socket to connect
0923 17:29:34.228064 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:29:34.228189 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:29:34.228545 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:29:34.228548 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:29:34.228568 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:29:34.228899 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.228911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.230004 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.230008 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.231521 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:29:34.231534 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.232801 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.232808 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:29:34.232986 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.233022 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:29:34.233130 (+ 108us) client_negotiation.cc:240] Negotiation successful
0923 17:29:34.233272 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1467,"mutex_wait_us":1,"thread_start_us":1277,"threads_started":1}
W20260923 17:29:34.234637 19110 tablet.cc:2369] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:29:34.234829 18931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caeeede503b45a1a3381c2227362320" candidate_uuid: "09001058d9d74a96a94ad5b4daf09594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76bb6b0f65f47959846462bea6021a5" is_pre_election: true
I20260923 17:29:34.234959 18797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caeeede503b45a1a3381c2227362320" candidate_uuid: "09001058d9d74a96a94ad5b4daf09594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515f3f9c129d4782be2551d43f1c783c" is_pre_election: true
I20260923 17:29:34.235447 18797 raft_consensus.cc:2454] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09001058d9d74a96a94ad5b4daf09594 in term 0.
I20260923 17:29:34.235445 18931 raft_consensus.cc:2454] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09001058d9d74a96a94ad5b4daf09594 in term 0.
I20260923 17:29:34.236114 18997 leader_election.cc:302] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09001058d9d74a96a94ad5b4daf09594, 515f3f9c129d4782be2551d43f1c783c; no voters:
I20260923 17:29:34.236491 19134 raft_consensus.cc:2781] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:29:34.236617 19134 raft_consensus.cc:482] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:29:34.236717 19134 raft_consensus.cc:3037] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:34.239094 19134 raft_consensus.cc:504] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } }
I20260923 17:29:34.239715 19134 leader_election.cc:288] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [CANDIDATE]: Term 1 election: Requested vote from peers 515f3f9c129d4782be2551d43f1c783c (127.8.29.193:35275), f76bb6b0f65f47959846462bea6021a5 (127.8.29.194:36441)
I20260923 17:29:34.240252 18797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caeeede503b45a1a3381c2227362320" candidate_uuid: "09001058d9d74a96a94ad5b4daf09594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "515f3f9c129d4782be2551d43f1c783c"
I20260923 17:29:34.240267 18931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1caeeede503b45a1a3381c2227362320" candidate_uuid: "09001058d9d74a96a94ad5b4daf09594" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f76bb6b0f65f47959846462bea6021a5"
I20260923 17:29:34.240491 18797 raft_consensus.cc:3037] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:34.240491 18931 raft_consensus.cc:3037] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:29:34.243703 18797 raft_consensus.cc:2454] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09001058d9d74a96a94ad5b4daf09594 in term 1.
I20260923 17:29:34.243703 18931 raft_consensus.cc:2454] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09001058d9d74a96a94ad5b4daf09594 in term 1.
I20260923 17:29:34.244177 19000 leader_election.cc:302] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09001058d9d74a96a94ad5b4daf09594, f76bb6b0f65f47959846462bea6021a5; no voters:
I20260923 17:29:34.244474 19134 raft_consensus.cc:2781] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:29:34.244925 19134 raft_consensus.cc:686] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 1 LEADER]: Becoming Leader. State: Replica: 09001058d9d74a96a94ad5b4daf09594, State: Running, Role: LEADER
I20260923 17:29:34.245397 19134 consensus_queue.cc:237] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } }
I20260923 17:29:34.250168 18652 catalog_manager.cc:5495] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09001058d9d74a96a94ad5b4daf09594 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "09001058d9d74a96a94ad5b4daf09594" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09001058d9d74a96a94ad5b4daf09594" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 37175 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 } health_report { overall_health: UNKNOWN } } }
I20260923 17:29:34.276149 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.270716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36204 (local address 127.8.29.195:37175)
0923 17:29:34.270909 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.270916 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.270984 (+ 68us) server_negotiation.cc:407] Connection header received
0923 17:29:34.271237 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.271241 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.271319 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.271449 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:29:34.271826 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.272524 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.273867 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.274282 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.274394 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.274988 (+ 594us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:29:34.275528 (+ 540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:29:34.275679 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.275796 (+ 117us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.275887 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:29:34.285521 18797 raft_consensus.cc:1261] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [term 1 FOLLOWER]: Refusing update from remote peer 09001058d9d74a96a94ad5b4daf09594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:34.285522 18931 raft_consensus.cc:1261] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [term 1 FOLLOWER]: Refusing update from remote peer 09001058d9d74a96a94ad5b4daf09594: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:29:34.286844 19134 consensus_queue.cc:1035] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "515f3f9c129d4782be2551d43f1c783c" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 35275 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:34.287271 19137 consensus_queue.cc:1035] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f76bb6b0f65f47959846462bea6021a5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 36441 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:29:34.299202 19140 mvcc.cc:205] Tried to move back new op lower bound from 7332596016260407296 to 7332596016112955392. Current Snapshot: MvccSnapshot[applied={T|T < 7332596016260407296}]
I20260923 17:29:34.304154 19142 mvcc.cc:205] Tried to move back new op lower bound from 7332596016260407296 to 7332596016112955392. Current Snapshot: MvccSnapshot[applied={T|T < 7332596016260407296}]
I20260923 17:29:34.314033 18652 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:34.314337 18652 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:29:34.318161 18652 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5150ea7d8ae44d92909f5aba4ceb219f: Sending DeleteTablet for 3 replicas of tablet 1caeeede503b45a1a3381c2227362320
I20260923 17:29:34.319474 19044 tablet_service.cc:1514] Processing DeleteTablet for tablet 1caeeede503b45a1a3381c2227362320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:34 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:36188
I20260923 17:29:34.319483 18777 tablet_service.cc:1514] Processing DeleteTablet for tablet 1caeeede503b45a1a3381c2227362320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:34 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:56906
I20260923 17:29:34.319659 18911 tablet_service.cc:1514] Processing DeleteTablet for tablet 1caeeede503b45a1a3381c2227362320 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:29:34 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:41420
I20260923 17:29:34.323096 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 18714
I20260923 17:29:34.331184 19151 tablet_replica.cc:330] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: stopping tablet replica
I20260923 17:29:34.331926 19150 tablet_replica.cc:330] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: stopping tablet replica
I20260923 17:29:34.332389 19151 raft_consensus.cc:2229] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:34.332474 19150 raft_consensus.cc:2229] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:29:34.333021 19151 raft_consensus.cc:2258] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:34.333336 19150 raft_consensus.cc:2258] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:29:34.334764 19152 tablet_replica.cc:330] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: stopping tablet replica
I20260923 17:29:34.335477 19152 raft_consensus.cc:2229] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:29:34.335994 19150 ts_tablet_manager.cc:1905] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:34.337337 19151 ts_tablet_manager.cc:1905] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:34.340663 19152 raft_consensus.cc:2258] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5 [term 1 FOLLOWER]: Raft consensus is shut down!
Sep 23 17:29:34 dist-test-slave-p7jt krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:29:34.343497 19152 ts_tablet_manager.cc:1905] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:29:34.343758 19150 ts_tablet_manager.cc:1918] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:29:34.344936 19151 ts_tablet_manager.cc:1918] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:29:34.345177 19151 log.cc:1196] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/1caeeede503b45a1a3381c2227362320
I20260923 17:29:34.345774 19151 ts_tablet_manager.cc:1939] T 1caeeede503b45a1a3381c2227362320 P 515f3f9c129d4782be2551d43f1c783c: Deleting consensus metadata
I20260923 17:29:34.347892 18637 catalog_manager.cc:4841] TS 515f3f9c129d4782be2551d43f1c783c (127.8.29.193:35275): tablet 1caeeede503b45a1a3381c2227362320 (table test-table [id=d37b2de565bc4d3e961d04a2bbe7a655]) successfully deleted
I20260923 17:29:34.348548 19150 log.cc:1196] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/1caeeede503b45a1a3381c2227362320
I20260923 17:29:34.349293 19150 ts_tablet_manager.cc:1939] T 1caeeede503b45a1a3381c2227362320 P 09001058d9d74a96a94ad5b4daf09594: Deleting consensus metadata
I20260923 17:29:34.350078 19123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.325137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56918 (local address 127.8.29.193:35275)
0923 17:29:34.325318 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.325325 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.328228 (+ 2903us) server_negotiation.cc:407] Connection header received
0923 17:29:34.328437 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.328443 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.328546 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.328663 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:34.332587 (+ 3924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.333833 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.336033 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.336736 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.336850 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.342392 (+ 5542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:34.342410 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:34.342432 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:34.342501 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:34.346405 (+ 3904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.347413 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.347423 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.347426 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.347623 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.348109 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.348114 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.348116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.348560 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:34.348869 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.349463 (+ 594us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.349718 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:29:34.350577 19152 ts_tablet_manager.cc:1918] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:29:34.350814 19152 log.cc:1196] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/1caeeede503b45a1a3381c2227362320
I20260923 17:29:34.351188 18639 catalog_manager.cc:4841] TS 09001058d9d74a96a94ad5b4daf09594 (127.8.29.195:37175): tablet 1caeeede503b45a1a3381c2227362320 (table test-table [id=d37b2de565bc4d3e961d04a2bbe7a655]) successfully deleted
I20260923 17:29:34.351378 19152 ts_tablet_manager.cc:1939] T 1caeeede503b45a1a3381c2227362320 P f76bb6b0f65f47959846462bea6021a5: Deleting consensus metadata
I20260923 17:29:34.352177 18837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:34.353266 18640 catalog_manager.cc:4841] TS f76bb6b0f65f47959846462bea6021a5 (127.8.29.194:36441): tablet 1caeeede503b45a1a3381c2227362320 (table test-table [id=d37b2de565bc4d3e961d04a2bbe7a655]) successfully deleted
I20260923 17:29:34.441388 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 18714
I20260923 17:29:34.456110 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 18848
Sep 23 17:29:34 dist-test-slave-p7jt krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:29:34.472568 19126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.456859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41434 (local address 127.8.29.194:36441)
0923 17:29:34.457028 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.457033 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.457051 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:29:34.457224 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.457228 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.457302 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.457375 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:34.458929 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.459664 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.461450 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.461838 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.461939 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.466853 (+ 4914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:34.466871 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:34.466888 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:34.466926 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:34.469727 (+ 2801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.470593 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.470601 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.470603 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.470744 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.471101 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.471104 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.471106 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.471477 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:34.471691 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.472086 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.472255 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:29:34.473263 18971 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:34.567756 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 18848
I20260923 17:29:34.581241 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 18981
Sep 23 17:29:34 dist-test-slave-p7jt krb5kdc[18608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184571, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:29:34.597172 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:29:34.582010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36210 (local address 127.8.29.195:37175)
0923 17:29:34.582172 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:29:34.582176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:29:34.582194 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:29:34.582399 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:29:34.582404 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:29:34.582567 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:29:34.582659 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:29:34.584384 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.585226 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.586494 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:29:34.586933 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:29:34.587044 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:29:34.591048 (+ 4004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:29:34.591066 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:29:34.591084 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:29:34.591132 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:29:34.594204 (+ 3072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.595106 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.595113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.595116 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.595276 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:29:34.595653 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:29:34.595657 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:29:34.595659 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:29:34.596028 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:29:34.596327 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0923 17:29:34.596628 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 17:29:34.596833 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:29:34.597826 19104 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:34.716578 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 18981
I20260923 17:29:34.731977 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 18621
I20260923 17:29:34.733170 18683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:29:34.840590 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 18621
2026-09-23T17:29:34Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4168 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:29:34 dist-test-slave-p7jt krb5kdc[19160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:29:34 dist-test-slave-p7jt krb5kdc[19160](info): set up 2 sockets
Sep 23 17:29:34 dist-test-slave-p7jt krb5kdc[19160](info): commencing operation
krb5kdc: starting...
W20260923 17:29:35.859196 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.980s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:29:36 dist-test-slave-p7jt krb5kdc[19160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:29:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:29:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.29.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.29.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/rangeradmin_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/rangeradmin_127.8.29.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.29.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.29.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/rangerlookup_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/rangerlookup_127.8.29.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.29.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.29.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/HTTP_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/HTTP_127.8.29.212@KRBTEST.COM.keytab.
I20260923 17:29:36.185554 8311 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Gj89p/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/postgres -l logfile start
2026-09-23 17:29:38.883 UTC [19190] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:29:38.883 UTC [19190] LOG: listening on IPv4 address "127.8.29.212", port 49173
2026-09-23 17:29:38.887 UTC [19190] LOG: listening on Unix socket "/tmp/.s.PGSQL.49173"
2026-09-23 17:29:38.907 UTC [19193] LOG: database system was shut down at 2026-09-23 17:29:37 UTC
2026-09-23 17:29:38.911 UTC [19190] LOG: database system is ready to accept connections
I20260923 17:29:39.806885 8311 mini_postgres.cc:96] Postgres bound to 49173
2026-09-23 17:29:39.823 UTC [19201] FATAL: database "slave" does not exist
127.8.29.212:49173 - accepting connections
I20260923 17:29:39.824515 8311 mini_ranger.cc:161] Starting Ranger...
I20260923 17:29:39.850896 8311 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:29:40.415921 8311 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:29:40.416074 8311 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-admin
2026-09-23 17:29:40,804 [I] DB FLAVOR :POSTGRES
2026-09-23 17:29:40,805 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:29:40,805 [I] Checking connection..
2026-09-23 17:29:40,805 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:29:41,079 [I] Checking connection passed.
2026-09-23 17:29:41,079 [I] --------- Verifying version history table ---------
2026-09-23 17:29:41,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:29:41,349 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:29:41,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:29:41,620 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:29:41,621 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:29:41,621 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:29:41.870 UTC [19265] WARNING: there is no transaction in progress
2026-09-23 17:29:41,877 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:29:41,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:29:42,153 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:29:42,153 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:29:42,154 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:29:42,414 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:29:42,652 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p7jt.c.gcp-upstream.internal','N') ;"
2026-09-23 17:29:42,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:29:43,126 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:29:43,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:29:43,360 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:29:43,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:29:43,622 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:29:43,851 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:29:43,851 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:29:44.927 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:44.952 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:44.976 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:44.995 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.018 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.039 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.105 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.115 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.131 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.142 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.152 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.169 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.181 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.187 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.196 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45.204 UTC [19373] WARNING: there is no transaction in progress
2026-09-23 17:29:45,988 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:29:45,988 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:29:46,231 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:29:46,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:29:46,496 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:29:46,496 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:29:46,745 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:29:46,745 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:29:47,005 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:29:47,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p7jt.c.gcp-upstream.internal';"
2026-09-23 17:29:47,484 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:29:47,485 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:29:47,748 [I] DB_PATCHES have already been applied
I20260923 17:29:47.754032 8311 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Gj89p/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:29:47.754146 8311 mini_ranger.cc:193] Using host: 127.8.29.212
I20260923 17:29:47.769409 8311 mini_ranger.cc:227] Ranger admin URL: http://127.8.29.212:36901
Sep 23, 2026 5:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:29:48 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:29:48.812172 8311 mini_ranger.cc:160] Time spent starting Ranger: real 8.988s user 0.003s sys 0.040s
Sep 23, 2026 5:30:09 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:30:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 5:30:09 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:30:09 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:30:10 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:30:11.568224 8311 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.29.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.29.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/rangerkms_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/rangerkms_127.8.29.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.29.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.29.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.29.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/HTTP_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.29.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/HTTP_127.8.29.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:30:11 dist-test-slave-p7jt krb5kdc[19160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184611, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:30:11.757159 8311 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:30:11.782867 8311 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:30:12.817621 8311 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:30:12.817754 8311 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-kms
I20260923 17:30:12.817914 8311 mini_ranger_kms.cc:227] Using postgres at 127.8.29.212:49173
2026-09-23 17:30:13,375 [I] DB FLAVOR :POSTGRES
2026-09-23 17:30:13,376 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:30:13,376 [I] Checking connection
2026-09-23 17:30:13,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:30:13,616 [I] connection success
2026-09-23 17:30:13,616 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:30:13,616 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:30:13,616 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:30:13,860 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:30:13,860 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:30:13,861 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:30:13,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:49173/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:30:14,141 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:30:14.648212 8311 mini_ranger_kms.cc:310] Created kms service
I20260923 17:30:14.906811 8311 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:30:14.957973 8311 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:30:14 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184611, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:30:15.368453 8311 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:30:15.368742 8311 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:30:15.368789 8311 mini_ranger_kms.cc:240] Using host: 127.8.29.212
I20260923 17:30:15.386376 8311 mini_ranger_kms.cc:276] Ranger KMS PID: 19554
I20260923 17:30:15.386653 8311 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.29.212:53263
Sep 23, 2026 5:30:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:30:16 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:30:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:30:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:30:16 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:30:16 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 17:30:17.436172 8311 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.679s user 0.111s sys 0.358s
I20260923 17:30:17.436446 8311 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:30:17.436684 8311 mini_ranger_kms.cc:401] 127.8.29.212:53263/kms/v1/keys
Sep 23 17:30:17 dist-test-slave-p7jt krb5kdc[19160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184617, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.29.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 17:30:17 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184617, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.29.212@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
Sep 23 17:30:18 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184617, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.29.212@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
Sep 23 17:30:20 dist-test-slave-p7jt krb5kdc[19160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:30:20.700088 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:33931
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:33931
--ranger_config_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:53263/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:30:21.041961 19604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:30:21.042255 19604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:30:21.042368 19604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:30:21.049758 19604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:30:21.049877 19604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:30:21.049948 19604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:30:21.050020 19604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:30:21.050082 19604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:30:21.060349 19604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.29.212:53263/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:33931
--ranger_config_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:33931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:30:21.061715 19604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:30:21.063114 19604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:30:21.077003 19609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:30:21.077049 19610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:30:21.077095 19604 server_base.cc:1034] running on GCE node
W20260923 17:30:21.077201 19612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:30:21.078393 19604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:30:21.079959 19604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:30:21.081167 19604 hybrid_clock.cc:648] HybridClock initialized: now 1790184621081117 us; error 45 us; skew 500 ppm
I20260923 17:30:21.081521 19604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:30:21 dist-test-slave-p7jt krb5kdc[19160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184621, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:30:21.091845 19604 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:30:21.093544 19604 webserver.cc:460] Webserver started at http://127.8.29.254:45613/ using document root <none> and password file <none>
I20260923 17:30:21.094211 19604 fs_manager.cc:359] Metadata directory not provided
I20260923 17:30:21.094316 19604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:30:21.094653 19604 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:30:21 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184621, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:30:21.143664 19604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7"
format_stamp: "Formatted at 2026-09-23 17:30:21 on dist-test-slave-p7jt"
server_key: "d74c707d093654111fe57d9dd1aeb73a"
server_key_iv: "8d20c112ef0073470ea3139de5ec391b"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:21.144495 19604 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7"
format_stamp: "Formatted at 2026-09-23 17:30:21 on dist-test-slave-p7jt"
server_key: "d74c707d093654111fe57d9dd1aeb73a"
server_key_iv: "8d20c112ef0073470ea3139de5ec391b"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:21.150121 19604 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:30:21.166733 19619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:21.168286 19604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:30:21.168465 19604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7"
format_stamp: "Formatted at 2026-09-23 17:30:21 on dist-test-slave-p7jt"
server_key: "d74c707d093654111fe57d9dd1aeb73a"
server_key_iv: "8d20c112ef0073470ea3139de5ec391b"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:21.168682 19604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:30:21.183175 19604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:30:21.187227 19604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:30:21.187498 19604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:30:21.206298 19604 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:33931
I20260923 17:30:21.206301 19670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:33931 every 8 connection(s)
I20260923 17:30:21.207692 19604 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:30:21.211767 19671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:30:21.216499 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 19604
I20260923 17:30:21.216755 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:30:21.217700 8311 external_mini_cluster.cc:1428] Setting key fd665a57231c7e3b35cf57b7fb849d10
I20260923 17:30:21.224777 19671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7: Bootstrap starting.
I20260923 17:30:21.228394 19671 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7: Neither blocks nor log segments found. Creating new log.
I20260923 17:30:21.229597 19671 log.cc:824] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:30:21 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:30:21.233336 19671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7: No bootstrap required, opened a new log
I20260923 17:30:21.241487 19674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:21.220164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36580 (local address 127.8.29.254:33931)
0923 17:30:21.221036 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:21.221050 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:21.221095 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:30:21.223127 (+ 2032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:21.223156 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:21.223640 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:21.224052 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:30:21.225527 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:21.226978 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:21.228298 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:21.229093 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:21.229274 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:21.233796 (+ 4522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:30:21.233806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:30:21.233824 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:30:21.233868 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:30:21.237180 (+ 3312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:21.237968 (+ 788us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:21.237975 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:21.237982 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:21.238135 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:21.238545 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:21.238549 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:21.238551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:21.239247 (+ 696us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:30:21.239558 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:21.240023 (+ 465us) server_negotiation.cc:299] Negotiation successful
0923 17:30:21.240451 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":369,"threads_started":1}
I20260923 17:30:21.241667 19671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33931 } }
I20260923 17:30:21.242141 19671 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:30:21.242256 19671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c543dc2f4f8440aa7174bbdae9e5ee7, State: Initialized, Role: FOLLOWER
I20260923 17:30:21.242937 19671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33931 } }
I20260923 17:30:21.243214 19671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:30:21.243363 19671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:30:21.243525 19671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:30:21.245970 19671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33931 } }
I20260923 17:30:21.246526 19671 leader_election.cc:302] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c543dc2f4f8440aa7174bbdae9e5ee7; no voters:
I20260923 17:30:21.247123 19671 leader_election.cc:288] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:30:21.247277 19676 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:30:21.248020 19676 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [term 1 LEADER]: Becoming Leader. State: Replica: 8c543dc2f4f8440aa7174bbdae9e5ee7, State: Running, Role: LEADER
I20260923 17:30:21.248570 19676 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33931 } }
I20260923 17:30:21.249186 19671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:30:21.251328 19678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c543dc2f4f8440aa7174bbdae9e5ee7. Latest consensus state: current_term: 1 leader_uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33931 } } }
I20260923 17:30:21.251842 19678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:30:21.251649 19677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c543dc2f4f8440aa7174bbdae9e5ee7" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33931 } } }
I20260923 17:30:21.252113 19677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7 [sys.catalog]: This master's current role is: LEADER
I20260923 17:30:21.252890 19684 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:30:21.259315 19684 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:30:21.269595 19684 catalog_manager.cc:1347] Generated new cluster ID: 3b552ca4648540b088142595c9432ca7
I20260923 17:30:21.269721 19684 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:30:21.274149 19671 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:30:21.288125 19684 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:30:21.290020 19684 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:30:21.302995 19684 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7: Generated new TSK 0
I20260923 17:30:21.304482 19684 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:30:22.059998 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:53263/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33931
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:30:22 dist-test-slave-p7jt krb5kdc[19160](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184622, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 17:30:22.468497 19716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:30:22.468794 19716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:30:22.468954 19716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:30:22.477240 19716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:30:22.477370 19716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:30:22.477526 19716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:30:22.489029 19716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.29.212:53263/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:30:22.490803 19716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:30:22.492094 19716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:30:22.508282 19726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:30:22.509123 19727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:30:22.516290 19729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:30:22.516454 19716 server_base.cc:1034] running on GCE node
I20260923 17:30:22.517282 19716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:30:22.518385 19716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:30:22.525920 19716 hybrid_clock.cc:648] HybridClock initialized: now 1790184622525837 us; error 53 us; skew 500 ppm
I20260923 17:30:22.526351 19716 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:30:22 dist-test-slave-p7jt krb5kdc[19160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184622, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:30:22.529793 19716 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:30:22.540540 19716 webserver.cc:460] Webserver started at http://127.8.29.193:40653/ using document root <none> and password file <none>
I20260923 17:30:22.541466 19716 fs_manager.cc:359] Metadata directory not provided
I20260923 17:30:22.541620 19716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:30:22.542033 19716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:30:22 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184622, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:30:22.561997 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "add950aa4dcd4c618ed449a578075482"
format_stamp: "Formatted at 2026-09-23 17:30:22 on dist-test-slave-p7jt"
server_key: "109dc6d3a31196a7bb46b7d6f3ee769c"
server_key_iv: "4a60f12b55c5ed48083bfb7be91fc15a"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:22.563445 19716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "add950aa4dcd4c618ed449a578075482"
format_stamp: "Formatted at 2026-09-23 17:30:22 on dist-test-slave-p7jt"
server_key: "109dc6d3a31196a7bb46b7d6f3ee769c"
server_key_iv: "4a60f12b55c5ed48083bfb7be91fc15a"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:22.576808 19716 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.010s sys 0.001s
I20260923 17:30:22.604768 19736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:22.606546 19716 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.004s sys 0.001s
I20260923 17:30:22.606745 19716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "add950aa4dcd4c618ed449a578075482"
format_stamp: "Formatted at 2026-09-23 17:30:22 on dist-test-slave-p7jt"
server_key: "109dc6d3a31196a7bb46b7d6f3ee769c"
server_key_iv: "4a60f12b55c5ed48083bfb7be91fc15a"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:22.606974 19716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:30:22.651875 19716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:30:22.656953 19716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:30:22.657256 19716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:30:22.658419 19716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:30:22.660420 19716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:30:22.660526 19716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:22.660694 19716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:30:22.660758 19716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:22.727638 19716 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:38697
I20260923 17:30:22.727638 19848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:38697 every 8 connection(s)
I20260923 17:30:22.729239 19716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:30:22.733548 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 19716
I20260923 17:30:22.733769 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:30:22.734620 8311 external_mini_cluster.cc:1428] Setting key 3ab7ecf9893bbc8d916c9dfcd9c45cb6
Sep 23 17:30:22 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184622, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:30:22.789621 19851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:22.732997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:55847 (local address 127.8.29.254:33931)
0923 17:30:22.759704 (+ 26707us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:22.759710 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:22.759738 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:30:22.767046 (+ 7308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:22.767053 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:22.767160 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:22.767278 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:30:22.769873 (+ 2595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:22.771065 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:22.773755 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:22.774428 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:22.774583 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:22.779843 (+ 5260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:30:22.779852 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:30:22.779858 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:30:22.779920 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:30:22.784629 (+ 4709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:22.786335 (+ 1706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:22.786342 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:22.786346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:22.786571 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:22.787208 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:22.787214 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:22.787217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:22.787593 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:30:22.787726 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:22.788925 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0923 17:30:22.789200 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26559,"thread_start_us":26366,"threads_started":1}
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:30:22.793602 19852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:22.733355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33931 (local address 127.8.29.193:55847)
0923 17:30:22.749964 (+ 16609us) negotiation.cc:107] Waiting for socket to connect
0923 17:30:22.750049 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0923 17:30:22.758770 (+ 8721us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:30:22.767871 (+ 9101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:30:22.767888 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:30:22.768627 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:30:22.769607 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:22.769641 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:22.771281 (+ 1640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:22.771286 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:22.773514 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:22.773557 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:22.774890 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:22.774899 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:22.775229 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:22.776039 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:30:22.776071 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:30:22.779473 (+ 3402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:30:22.785397 (+ 5924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:30:22.785410 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:30:22.785433 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:30:22.786153 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:30:22.786754 (+ 601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:30:22.786760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:30:22.786764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:30:22.787067 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:30:22.787855 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:30:22.787865 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:30:22.788377 (+ 512us) client_negotiation.cc:769] Sending connection context
0923 17:30:22.789986 (+ 1609us) client_negotiation.cc:240] Negotiation successful
0923 17:30:22.790693 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16203,"thread_start_us":17023,"threads_started":1}
I20260923 17:30:22.795938 19849 heartbeater.cc:347] Connected to a master server at 127.8.29.254:33931
I20260923 17:30:22.796329 19849 heartbeater.cc:464] Registering TS with master...
I20260923 17:30:22.797250 19849 heartbeater.cc:511] Master 127.8.29.254:33931 requested a full tablet report, sending...
I20260923 17:30:22.800209 19636 ts_manager.cc:194] Registered new tserver with Master: add950aa4dcd4c618ed449a578075482 (127.8.29.193:38697)
I20260923 17:30:22.802450 19636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:55847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:30:22.839324 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:53263/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33931
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:30:22 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184622, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
Sep 23 17:30:23 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184622, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
W20260923 17:30:23.377612 19855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:30:23.378120 19855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:30:23.378314 19855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:30:23.390166 19855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:30:23.390417 19855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:30:23.390677 19855 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:30:23.406111 19855 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.29.212:53263/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:30:23.408236 19855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:30:23.410109 19855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:30:23.464607 19863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:30:23.464623 19862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:30:23.479081 19855 server_base.cc:1034] running on GCE node
W20260923 17:30:23.479287 19866 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:30:23.480226 19855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:30:23.484823 19855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:30:23.497216 19855 hybrid_clock.cc:648] HybridClock initialized: now 1790184623497027 us; error 175 us; skew 500 ppm
I20260923 17:30:23.497807 19855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:30:23 dist-test-slave-p7jt krb5kdc[19160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184623, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:30:23.505740 19855 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:30:23.535027 19855 webserver.cc:460] Webserver started at http://127.8.29.194:35099/ using document root <none> and password file <none>
I20260923 17:30:23.536362 19855 fs_manager.cc:359] Metadata directory not provided
I20260923 17:30:23.536628 19855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:30:23.537277 19855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:30:23 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184623, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:30:23.581185 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "06e351aab20b464e97bd992d6429c611"
format_stamp: "Formatted at 2026-09-23 17:30:23 on dist-test-slave-p7jt"
server_key: "1c9396ceaaf02911483ce2a275af1def"
server_key_iv: "e60e7d038564fbbb440acccad8d74b6d"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:23.582288 19855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "06e351aab20b464e97bd992d6429c611"
format_stamp: "Formatted at 2026-09-23 17:30:23 on dist-test-slave-p7jt"
server_key: "1c9396ceaaf02911483ce2a275af1def"
server_key_iv: "e60e7d038564fbbb440acccad8d74b6d"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:23.602080 19855 fs_manager.cc:691] Time spent creating directory manager: real 0.019s user 0.013s sys 0.004s
I20260923 17:30:23.631393 19879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:23.633690 19855 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.008s sys 0.000s
I20260923 17:30:23.633895 19855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "06e351aab20b464e97bd992d6429c611"
format_stamp: "Formatted at 2026-09-23 17:30:23 on dist-test-slave-p7jt"
server_key: "1c9396ceaaf02911483ce2a275af1def"
server_key_iv: "e60e7d038564fbbb440acccad8d74b6d"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:23.634263 19855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:30:23.660277 19855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:30:23.665836 19855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:30:23.666198 19855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:30:23.667474 19855 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:30:23.669430 19855 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:30:23.669533 19855 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:23.669724 19855 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:30:23.669797 19855 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:23.715755 19855 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:40173
I20260923 17:30:23.715762 19991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:40173 every 8 connection(s)
I20260923 17:30:23.717577 19855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:30:23.717964 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 19855
I20260923 17:30:23.718134 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:30:23.718690 8311 external_mini_cluster.cc:1428] Setting key 36b9bce480da033b6216c8885f8537c5
Sep 23 17:30:23 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184623, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
I20260923 17:30:23.754537 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:23.723342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:37349 (local address 127.8.29.254:33931)
0923 17:30:23.725321 (+ 1979us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:23.725328 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:23.733651 (+ 8323us) server_negotiation.cc:407] Connection header received
0923 17:30:23.736487 (+ 2836us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:23.736492 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:23.736576 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:23.736670 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:30:23.738701 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:23.739594 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:23.741931 (+ 2337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:23.742546 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:23.742651 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:23.747289 (+ 4638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:30:23.747298 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:30:23.747304 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:30:23.747352 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:30:23.750400 (+ 3048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:23.751332 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:23.751338 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:23.751349 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:23.751542 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:23.752294 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:23.752299 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:23.752302 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:23.752693 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:30:23.752835 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:23.753877 (+ 1042us) server_negotiation.cc:299] Negotiation successful
0923 17:30:23.754135 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1853,"thread_start_us":1753,"threads_started":1}
I20260923 17:30:23.757095 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:23.722797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33931 (local address 127.8.29.194:37349)
0923 17:30:23.733382 (+ 10585us) negotiation.cc:107] Waiting for socket to connect
0923 17:30:23.733443 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 17:30:23.735995 (+ 2552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:30:23.736989 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:30:23.737001 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:30:23.737641 (+ 640us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:30:23.738519 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:23.738547 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:23.739786 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:23.739791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:23.741758 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:23.741774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:23.743031 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:23.743037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:23.743331 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:23.744095 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:30:23.744128 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:30:23.746965 (+ 2837us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:30:23.750624 (+ 3659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:30:23.750634 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:30:23.750652 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:30:23.751180 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:30:23.751817 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:30:23.751821 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:30:23.751825 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:30:23.752130 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:30:23.752937 (+ 807us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:30:23.752945 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:30:23.753431 (+ 486us) client_negotiation.cc:769] Sending connection context
0923 17:30:23.754961 (+ 1530us) client_negotiation.cc:240] Negotiation successful
0923 17:30:23.755572 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10156,"thread_start_us":10505,"threads_started":1}
I20260923 17:30:23.759025 19992 heartbeater.cc:347] Connected to a master server at 127.8.29.254:33931
I20260923 17:30:23.759339 19992 heartbeater.cc:464] Registering TS with master...
I20260923 17:30:23.760150 19992 heartbeater.cc:511] Master 127.8.29.254:33931 requested a full tablet report, sending...
I20260923 17:30:23.761977 19636 ts_manager.cc:194] Registered new tserver with Master: 06e351aab20b464e97bd992d6429c611 (127.8.29.194:40173)
I20260923 17:30:23.763123 19636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:37349
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:30:23.786054 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:53263/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33931
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:30:23.806933 19849 heartbeater.cc:503] Master 127.8.29.254:33931 was elected leader, sending a full tablet report...
W20260923 17:30:24.127568 19998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:30:24.127866 19998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:30:24.127998 19998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:30:24.135602 19998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:30:24.135741 19998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:30:24.135888 19998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:30:24.146308 19998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.29.212:53263/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33931
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:30:24.147692 19998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:30:24.149029 19998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:30:24.164762 20004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:30:24.164793 20006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:30:24.165050 20003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:30:24.165943 19998 server_base.cc:1034] running on GCE node
I20260923 17:30:24.166725 19998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:30:24.167929 19998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:30:24.169173 19998 hybrid_clock.cc:648] HybridClock initialized: now 1790184624169106 us; error 50 us; skew 500 ppm
I20260923 17:30:24.169559 19998 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:30:24 dist-test-slave-p7jt krb5kdc[19160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:30:24.173820 19998 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:30:24.175521 19998 webserver.cc:460] Webserver started at http://127.8.29.195:38021/ using document root <none> and password file <none>
I20260923 17:30:24.176234 19998 fs_manager.cc:359] Metadata directory not provided
I20260923 17:30:24.176337 19998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:30:24.176679 19998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:30:24 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:30:24.196197 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "8f0e1f11f75341359d72485367e26e26"
format_stamp: "Formatted at 2026-09-23 17:30:24 on dist-test-slave-p7jt"
server_key: "bc87ffac9b5a6db21b872fd9d57edfb8"
server_key_iv: "8d366d9846c7f146edba270e59ac3c77"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:24.197266 19998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "8f0e1f11f75341359d72485367e26e26"
format_stamp: "Formatted at 2026-09-23 17:30:24 on dist-test-slave-p7jt"
server_key: "bc87ffac9b5a6db21b872fd9d57edfb8"
server_key_iv: "8d366d9846c7f146edba270e59ac3c77"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:24.204581 19998 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260923 17:30:24.226318 20013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:24.228453 19998 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:30:24.228677 19998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "8f0e1f11f75341359d72485367e26e26"
format_stamp: "Formatted at 2026-09-23 17:30:24 on dist-test-slave-p7jt"
server_key: "bc87ffac9b5a6db21b872fd9d57edfb8"
server_key_iv: "8d366d9846c7f146edba270e59ac3c77"
server_key_version: "kuduclusterkey@0"
I20260923 17:30:24.228950 19998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:30:24.245170 19998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:30:24.249104 19998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:30:24.249409 19998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:30:24.250427 19998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:30:24.251922 19998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:30:24.251998 19998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:24.252107 19998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:30:24.252159 19998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:30:24.286010 19998 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:32797
I20260923 17:30:24.286036 20125 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:32797 every 8 connection(s)
I20260923 17:30:24.287338 19998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:30:24.291899 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 19998
I20260923 17:30:24.292093 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:30:24.292631 8311 external_mini_cluster.cc:1428] Setting key 96add586b170479831ad05f3ff54f592
Sep 23 17:30:24 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184624, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:30:24.309492 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.289875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:59375 (local address 127.8.29.254:33931)
0923 17:30:24.290380 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:24.290386 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:24.291446 (+ 1060us) server_negotiation.cc:407] Connection header received
0923 17:30:24.293604 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:24.293609 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:24.293699 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:24.293807 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:30:24.295944 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.296681 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.298975 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.299473 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.299590 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.303370 (+ 3780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:30:24.303375 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:30:24.303378 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:30:24.303412 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:30:24.305955 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:24.306962 (+ 1007us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:24.306968 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:24.306972 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:24.307148 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:24.307653 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:24.307659 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:24.307662 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:24.308005 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:30:24.308158 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:24.308895 (+ 737us) server_negotiation.cc:299] Negotiation successful
0923 17:30:24.309134 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":235,"threads_started":1}
I20260923 17:30:24.310388 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.290431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33931 (local address 127.8.29.195:59375)
0923 17:30:24.291251 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0923 17:30:24.291310 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:30:24.293179 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:30:24.294290 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:30:24.294303 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:30:24.294924 (+ 621us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:30:24.295766 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.295794 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.296834 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.296839 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.298795 (+ 1956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.298811 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.299706 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.299713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.300001 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.300720 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:30:24.300750 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:30:24.303124 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:30:24.306118 (+ 2994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:30:24.306128 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:30:24.306149 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:30:24.306794 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:30:24.307270 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:30:24.307275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:30:24.307279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:30:24.307532 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:30:24.308144 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:30:24.308154 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:30:24.308505 (+ 351us) client_negotiation.cc:769] Sending connection context
0923 17:30:24.308759 (+ 254us) client_negotiation.cc:240] Negotiation successful
0923 17:30:24.309170 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":317,"threads_started":1}
I20260923 17:30:24.311954 20126 heartbeater.cc:347] Connected to a master server at 127.8.29.254:33931
I20260923 17:30:24.312258 20126 heartbeater.cc:464] Registering TS with master...
I20260923 17:30:24.313040 20126 heartbeater.cc:511] Master 127.8.29.254:33931 requested a full tablet report, sending...
I20260923 17:30:24.314673 19636 ts_manager.cc:194] Registered new tserver with Master: 8f0e1f11f75341359d72485367e26e26 (127.8.29.195:32797)
I20260923 17:30:24.315810 19636 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:59375
I20260923 17:30:24.320811 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:30:24.339439 20128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.325587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36596 (local address 127.8.29.254:33931)
0923 17:30:24.325780 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:24.325786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:24.326111 (+ 325us) server_negotiation.cc:407] Connection header received
0923 17:30:24.326271 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:24.326276 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:24.326368 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:24.326493 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:30:24.327904 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.328977 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.330321 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.330993 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.331119 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.333368 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:30:24.333375 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:30:24.333380 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:30:24.333421 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:30:24.336655 (+ 3234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:24.337395 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:24.337400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:24.337403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:24.337564 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:24.337911 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:24.337915 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:24.337919 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:24.338305 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:30:24.338566 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:24.338867 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 17:30:24.339098 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:30:24.344165 19636 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36596:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:30:24.352195 19636 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:30:24.390838 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.366117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:40173 (local address 127.0.0.1:43394)
0923 17:30:24.378424 (+ 12307us) negotiation.cc:107] Waiting for socket to connect
0923 17:30:24.378502 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0923 17:30:24.378707 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:30:24.382736 (+ 4029us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:30:24.382744 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:30:24.382777 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:30:24.383182 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.383195 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.385264 (+ 2069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.385269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.387614 (+ 2345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.387632 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.389798 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.389805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.390018 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.390085 (+ 67us) client_negotiation.cc:769] Sending connection context
0923 17:30:24.390214 (+ 129us) client_negotiation.cc:240] Negotiation successful
0923 17:30:24.390432 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12015,"threads_started":1}
I20260923 17:30:24.392257 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.366295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.8.29.194:40173)
0923 17:30:24.381729 (+ 15434us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:24.381739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:24.381771 (+ 32us) server_negotiation.cc:407] Connection header received
0923 17:30:24.381876 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:24.381884 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:24.382192 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:24.382411 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:30:24.383584 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.385103 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.387833 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.389272 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.389398 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.389615 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:24.391699 (+ 2084us) server_negotiation.cc:299] Negotiation successful
0923 17:30:24.391903 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15260,"thread_start_us":17030,"threads_started":1}
I20260923 17:30:24.400223 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.366838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41594 (local address 127.8.29.193:38697)
0923 17:30:24.379057 (+ 12219us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:24.379068 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:24.391491 (+ 12423us) server_negotiation.cc:407] Connection header received
0923 17:30:24.391667 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:24.391677 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:24.391996 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:24.392244 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:30:24.392943 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.394814 (+ 1871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.397234 (+ 2420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.398757 (+ 1523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.398863 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.399104 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:24.399694 (+ 590us) server_negotiation.cc:299] Negotiation successful
0923 17:30:24.399876 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12009,"thread_start_us":11802,"threads_started":1}
I20260923 17:30:24.401130 20140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.366148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:38697 (local address 127.0.0.1:41594)
0923 17:30:24.391356 (+ 25208us) negotiation.cc:107] Waiting for socket to connect
0923 17:30:24.391377 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:30:24.391507 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:30:24.392508 (+ 1001us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:30:24.392512 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:30:24.392540 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:30:24.392768 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.392780 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.395065 (+ 2285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.395071 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.397016 (+ 1945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.397034 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.399244 (+ 2210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.399250 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.399450 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.399496 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 17:30:24.400707 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0923 17:30:24.400848 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25092,"thread_start_us":10925,"threads_started":1}
I20260923 17:30:24.406728 20143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.368070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:32797 (local address 127.0.0.1:45188)
0923 17:30:24.397244 (+ 29174us) negotiation.cc:107] Waiting for socket to connect
0923 17:30:24.397263 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:30:24.397381 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:30:24.398507 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:30:24.398513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:30:24.398540 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:30:24.398965 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.398975 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.401056 (+ 2081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.401062 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.403371 (+ 2309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.403389 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.405944 (+ 2555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.405950 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.406131 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.406164 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:30:24.406242 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:30:24.406377 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29054,"thread_start_us":35684,"threads_started":1}
I20260923 17:30:24.406860 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.368672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45188 (local address 127.8.29.195:32797)
0923 17:30:24.396620 (+ 27948us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:24.396630 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:24.397514 (+ 884us) server_negotiation.cc:407] Connection header received
0923 17:30:24.397632 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:24.397641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:24.397963 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:24.398197 (+ 234us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:30:24.399179 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.400864 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.403992 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.405456 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.405574 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.405808 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:24.406333 (+ 525us) server_negotiation.cc:299] Negotiation successful
0923 17:30:24.406543 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27748,"thread_start_us":27965,"threads_started":1}
I20260923 17:30:24.408915 19927 tablet_service.cc:1467] Processing CreateTablet for tablet f3c7befa37384b81a22a249f37f3f605 (DEFAULT_TABLE table=test-table [id=752ef9cbd7dc4c5fb3809a8ce671f39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:30:24.409773 20061 tablet_service.cc:1467] Processing CreateTablet for tablet f3c7befa37384b81a22a249f37f3f605 (DEFAULT_TABLE table=test-table [id=752ef9cbd7dc4c5fb3809a8ce671f39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:30:24.405839 19784 tablet_service.cc:1467] Processing CreateTablet for tablet f3c7befa37384b81a22a249f37f3f605 (DEFAULT_TABLE table=test-table [id=752ef9cbd7dc4c5fb3809a8ce671f39a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:30:24.410447 19927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3c7befa37384b81a22a249f37f3f605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:30:24.411371 20061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3c7befa37384b81a22a249f37f3f605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:30:24.411655 19784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3c7befa37384b81a22a249f37f3f605. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:30:24.431504 20145 tablet_bootstrap.cc:492] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: Bootstrap starting.
I20260923 17:30:24.434005 20146 tablet_bootstrap.cc:492] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: Bootstrap starting.
I20260923 17:30:24.437636 20145 tablet_bootstrap.cc:654] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: Neither blocks nor log segments found. Creating new log.
I20260923 17:30:24.438129 20146 tablet_bootstrap.cc:654] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: Neither blocks nor log segments found. Creating new log.
I20260923 17:30:24.439076 20147 tablet_bootstrap.cc:492] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: Bootstrap starting.
I20260923 17:30:24.439198 20145 log.cc:824] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: Log is configured to *not* fsync() on all Append() calls
I20260923 17:30:24.440853 20146 log.cc:824] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: Log is configured to *not* fsync() on all Append() calls
I20260923 17:30:24.443843 20147 tablet_bootstrap.cc:654] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: Neither blocks nor log segments found. Creating new log.
I20260923 17:30:24.445343 20147 log.cc:824] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: Log is configured to *not* fsync() on all Append() calls
I20260923 17:30:24.449321 20145 tablet_bootstrap.cc:492] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: No bootstrap required, opened a new log
I20260923 17:30:24.449321 20147 tablet_bootstrap.cc:492] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: No bootstrap required, opened a new log
I20260923 17:30:24.449317 20146 tablet_bootstrap.cc:492] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: No bootstrap required, opened a new log
I20260923 17:30:24.449635 20146 ts_tablet_manager.cc:1397] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20260923 17:30:24.449642 20145 ts_tablet_manager.cc:1397] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.010s
I20260923 17:30:24.449644 20147 ts_tablet_manager.cc:1397] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260923 17:30:24.459740 20146 raft_consensus.cc:348] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } }
I20260923 17:30:24.459973 20147 raft_consensus.cc:348] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } }
I20260923 17:30:24.460404 20146 raft_consensus.cc:374] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:30:24.460520 20147 raft_consensus.cc:374] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:30:24.460562 20146 raft_consensus.cc:729] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: add950aa4dcd4c618ed449a578075482, State: Initialized, Role: FOLLOWER
I20260923 17:30:24.460695 20147 raft_consensus.cc:729] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f0e1f11f75341359d72485367e26e26, State: Initialized, Role: FOLLOWER
I20260923 17:30:24.461380 20146 consensus_queue.cc:260] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } }
I20260923 17:30:24.461529 20147 consensus_queue.cc:260] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } }
I20260923 17:30:24.461731 20145 raft_consensus.cc:348] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } }
I20260923 17:30:24.462222 20145 raft_consensus.cc:374] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:30:24.462352 20145 raft_consensus.cc:729] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e351aab20b464e97bd992d6429c611, State: Initialized, Role: FOLLOWER
I20260923 17:30:24.463075 20145 consensus_queue.cc:260] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } }
I20260923 17:30:24.469924 20146 ts_tablet_manager.cc:1428] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: Time spent starting tablet: real 0.020s user 0.015s sys 0.003s
I20260923 17:30:24.469941 20147 ts_tablet_manager.cc:1428] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: Time spent starting tablet: real 0.020s user 0.013s sys 0.005s
I20260923 17:30:24.470386 20126 heartbeater.cc:503] Master 127.8.29.254:33931 was elected leader, sending a full tablet report...
I20260923 17:30:24.472843 19992 heartbeater.cc:503] Master 127.8.29.254:33931 was elected leader, sending a full tablet report...
I20260923 17:30:24.475414 20145 ts_tablet_manager.cc:1428] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: Time spent starting tablet: real 0.026s user 0.020s sys 0.002s
W20260923 17:30:24.477713 19993 tablet.cc:2369] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:30:24.484194 19850 tablet.cc:2369] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:30:24.538947 20127 tablet.cc:2369] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:30:24.745500 20152 raft_consensus.cc:482] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:30:24.745793 20152 raft_consensus.cc:504] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } }
I20260923 17:30:24.747491 20152 leader_election.cc:288] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers add950aa4dcd4c618ed449a578075482 (127.8.29.193:38697), 06e351aab20b464e97bd992d6429c611 (127.8.29.194:40173)
I20260923 17:30:24.753820 20129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.747887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:40173 (local address 127.8.29.195:35511)
0923 17:30:24.748271 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0923 17:30:24.748291 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:30:24.748460 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:30:24.748877 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:30:24.748883 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:30:24.748912 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:30:24.749137 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.749149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.750310 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.750316 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.751800 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.751822 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.753118 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.753124 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.753317 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.753349 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:30:24.753448 (+ 99us) client_negotiation.cc:240] Negotiation successful
0923 17:30:24.753601 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270}
I20260923 17:30:24.753885 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.748071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:35511 (local address 127.8.29.194:40173)
0923 17:30:24.748289 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:24.748294 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:24.748370 (+ 76us) server_negotiation.cc:407] Connection header received
0923 17:30:24.748643 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:24.748649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:24.748737 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:24.748825 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:30:24.749281 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.750166 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.751987 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.752938 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.753062 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.753118 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:24.753491 (+ 373us) server_negotiation.cc:299] Negotiation successful
0923 17:30:24.753633 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:30:24.754271 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.748052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:45155 (local address 127.8.29.193:38697)
0923 17:30:24.748260 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:24.748266 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:24.748805 (+ 539us) server_negotiation.cc:407] Connection header received
0923 17:30:24.748947 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:24.748952 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:24.749026 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:24.749143 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:30:24.749694 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.750595 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.752431 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.753354 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.753507 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.753586 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:24.753918 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:30:24.754055 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:30:24.754347 20154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.748095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:38697 (local address 127.8.29.195:45155)
0923 17:30:24.748678 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0923 17:30:24.748694 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:30:24.748823 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:30:24.749149 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:30:24.749153 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:30:24.749172 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:30:24.749509 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.749522 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.750738 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.750742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.752249 (+ 1507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:30:24.752269 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.753562 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.753568 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:30:24.753741 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.753775 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:30:24.753872 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:30:24.754036 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":287,"threads_started":1}
I20260923 17:30:24.755232 19947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3c7befa37384b81a22a249f37f3f605" candidate_uuid: "8f0e1f11f75341359d72485367e26e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e351aab20b464e97bd992d6429c611" is_pre_election: true
I20260923 17:30:24.755412 19804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3c7befa37384b81a22a249f37f3f605" candidate_uuid: "8f0e1f11f75341359d72485367e26e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add950aa4dcd4c618ed449a578075482" is_pre_election: true
I20260923 17:30:24.755846 19947 raft_consensus.cc:2454] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f0e1f11f75341359d72485367e26e26 in term 0.
I20260923 17:30:24.755885 19804 raft_consensus.cc:2454] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f0e1f11f75341359d72485367e26e26 in term 0.
I20260923 17:30:24.756592 20015 leader_election.cc:302] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e351aab20b464e97bd992d6429c611, 8f0e1f11f75341359d72485367e26e26; no voters:
I20260923 17:30:24.757011 20152 raft_consensus.cc:2781] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:30:24.757200 20152 raft_consensus.cc:482] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:30:24.757325 20152 raft_consensus.cc:3037] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:30:24.759791 20152 raft_consensus.cc:504] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } }
I20260923 17:30:24.760444 20152 leader_election.cc:288] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [CANDIDATE]: Term 1 election: Requested vote from peers add950aa4dcd4c618ed449a578075482 (127.8.29.193:38697), 06e351aab20b464e97bd992d6429c611 (127.8.29.194:40173)
I20260923 17:30:24.760901 19947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3c7befa37384b81a22a249f37f3f605" candidate_uuid: "8f0e1f11f75341359d72485367e26e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e351aab20b464e97bd992d6429c611"
I20260923 17:30:24.760931 19804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3c7befa37384b81a22a249f37f3f605" candidate_uuid: "8f0e1f11f75341359d72485367e26e26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "add950aa4dcd4c618ed449a578075482"
I20260923 17:30:24.761083 19947 raft_consensus.cc:3037] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:30:24.761124 19804 raft_consensus.cc:3037] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:30:24.763245 19947 raft_consensus.cc:2454] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f0e1f11f75341359d72485367e26e26 in term 1.
I20260923 17:30:24.763510 19804 raft_consensus.cc:2454] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f0e1f11f75341359d72485367e26e26 in term 1.
I20260923 17:30:24.763639 20015 leader_election.cc:302] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e351aab20b464e97bd992d6429c611, 8f0e1f11f75341359d72485367e26e26; no voters:
I20260923 17:30:24.763940 20152 raft_consensus.cc:2781] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:30:24.764397 20152 raft_consensus.cc:686] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 1 LEADER]: Becoming Leader. State: Replica: 8f0e1f11f75341359d72485367e26e26, State: Running, Role: LEADER
I20260923 17:30:24.764919 20152 consensus_queue.cc:237] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } }
I20260923 17:30:24.769680 19636 catalog_manager.cc:5495] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f0e1f11f75341359d72485367e26e26 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "8f0e1f11f75341359d72485367e26e26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8f0e1f11f75341359d72485367e26e26" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 32797 } health_report { overall_health: HEALTHY } } }
I20260923 17:30:24.843977 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.838380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.8.29.195:32797)
0923 17:30:24.838595 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:24.838599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:24.838633 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:30:24.838786 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:24.838790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:24.838874 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:24.838998 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:30:24.839363 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.840220 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.841811 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.842217 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.842316 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.842918 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:30:24.843484 (+ 566us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:30:24.843602 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:24.843701 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 17:30:24.843769 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:30:24.852061 19804 raft_consensus.cc:1261] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [term 1 FOLLOWER]: Refusing update from remote peer 8f0e1f11f75341359d72485367e26e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:30:24.852058 19947 raft_consensus.cc:1261] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [term 1 FOLLOWER]: Refusing update from remote peer 8f0e1f11f75341359d72485367e26e26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:30:24.853000 20152 consensus_queue.cc:1035] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "add950aa4dcd4c618ed449a578075482" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 38697 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:30:24.853284 20155 consensus_queue.cc:1035] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e351aab20b464e97bd992d6429c611" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 40173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:30:24.868791 20160 mvcc.cc:205] Tried to move back new op lower bound from 7332596223384387584 to 7332596223040958464. Current Snapshot: MvccSnapshot[applied={T|T < 7332596223384387584}]
I20260923 17:30:24.872071 20159 mvcc.cc:205] Tried to move back new op lower bound from 7332596223384387584 to 7332596223040958464. Current Snapshot: MvccSnapshot[applied={T|T < 7332596223384387584}]
I20260923 17:30:24.884431 19636 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:30:24.884680 19636 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36596:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:30:24.888411 19636 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c543dc2f4f8440aa7174bbdae9e5ee7: Sending DeleteTablet for 3 replicas of tablet f3c7befa37384b81a22a249f37f3f605
I20260923 17:30:24.889745 19927 tablet_service.cc:1514] Processing DeleteTablet for tablet f3c7befa37384b81a22a249f37f3f605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:30:24 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:43394
I20260923 17:30:24.890074 20061 tablet_service.cc:1514] Processing DeleteTablet for tablet f3c7befa37384b81a22a249f37f3f605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:30:24 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:45188
I20260923 17:30:24.890547 19784 tablet_service.cc:1514] Processing DeleteTablet for tablet f3c7befa37384b81a22a249f37f3f605 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:30:24 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:41594
I20260923 17:30:24.892325 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 19716
I20260923 17:30:24.900998 20170 tablet_replica.cc:330] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: stopping tablet replica
I20260923 17:30:24.901544 20170 raft_consensus.cc:2229] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:30:24.902139 20170 raft_consensus.cc:2258] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:30:24.903808 20169 tablet_replica.cc:330] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: stopping tablet replica
I20260923 17:30:24.904275 20169 raft_consensus.cc:2229] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:30:24.904953 20169 raft_consensus.cc:2258] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:30:24.906148 20168 tablet_replica.cc:330] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: stopping tablet replica
I20260923 17:30:24.906889 20170 ts_tablet_manager.cc:1905] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:30:24.906878 20168 raft_consensus.cc:2229] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:30:24.907393 20168 raft_consensus.cc:2258] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:30:24.909756 20168 ts_tablet_manager.cc:1905] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:30:24.909762 20169 ts_tablet_manager.cc:1905] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:30:24.914306 20170 ts_tablet_manager.cc:1918] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
Sep 23 17:30:24 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:30:24.916925 20169 ts_tablet_manager.cc:1918] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:30:24.916961 20168 ts_tablet_manager.cc:1918] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:30:24.917179 20168 log.cc:1196] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/f3c7befa37384b81a22a249f37f3f605
I20260923 17:30:24.917263 20169 log.cc:1196] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/f3c7befa37384b81a22a249f37f3f605
I20260923 17:30:24.917737 20168 ts_tablet_manager.cc:1939] T f3c7befa37384b81a22a249f37f3f605 P 06e351aab20b464e97bd992d6429c611: Deleting consensus metadata
I20260923 17:30:24.917888 20169 ts_tablet_manager.cc:1939] T f3c7befa37384b81a22a249f37f3f605 P 8f0e1f11f75341359d72485367e26e26: Deleting consensus metadata
I20260923 17:30:24.919706 20170 log.cc:1196] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/f3c7befa37384b81a22a249f37f3f605
I20260923 17:30:24.919778 19621 catalog_manager.cc:4841] TS 06e351aab20b464e97bd992d6429c611 (127.8.29.194:40173): tablet f3c7befa37384b81a22a249f37f3f605 (table test-table [id=752ef9cbd7dc4c5fb3809a8ce671f39a]) successfully deleted
I20260923 17:30:24.920413 20170 ts_tablet_manager.cc:1939] T f3c7befa37384b81a22a249f37f3f605 P add950aa4dcd4c618ed449a578075482: Deleting consensus metadata
I20260923 17:30:24.920421 19620 catalog_manager.cc:4841] TS 8f0e1f11f75341359d72485367e26e26 (127.8.29.195:32797): tablet f3c7befa37384b81a22a249f37f3f605 (table test-table [id=752ef9cbd7dc4c5fb3809a8ce671f39a]) successfully deleted
I20260923 17:30:24.922255 19623 catalog_manager.cc:4841] TS add950aa4dcd4c618ed449a578075482 (127.8.29.193:38697): tablet f3c7befa37384b81a22a249f37f3f605 (table test-table [id=752ef9cbd7dc4c5fb3809a8ce671f39a]) successfully deleted
I20260923 17:30:24.923947 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:24.894078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41596 (local address 127.8.29.193:38697)
0923 17:30:24.894275 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:24.894282 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:24.903119 (+ 8837us) server_negotiation.cc:407] Connection header received
0923 17:30:24.903320 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:24.903325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:24.903408 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:24.903510 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:30:24.907174 (+ 3664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.908223 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.910289 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:24.911000 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:24.911113 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:24.916857 (+ 5744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:30:24.916878 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:30:24.916900 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:30:24.916959 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:30:24.921094 (+ 4135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:24.921847 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:24.921854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:24.921856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:24.921998 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:24.922412 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:24.922416 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:24.922418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:24.922818 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:30:24.923106 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:24.923423 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:30:24.923623 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:30:24.924595 19844 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:30:25.014147 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 19716
I20260923 17:30:25.029222 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 19855
Sep 23 17:30:25 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:30:25.044800 20141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:25.029948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.8.29.194:40173)
0923 17:30:25.030158 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:25.030164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:25.030241 (+ 77us) server_negotiation.cc:407] Connection header received
0923 17:30:25.030424 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:25.030430 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:25.030549 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:25.030637 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:30:25.031912 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:25.032625 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:25.034097 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:25.034651 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:25.034766 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:25.038940 (+ 4174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:30:25.038957 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:30:25.038973 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:30:25.039010 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:30:25.041891 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:25.042811 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:25.042819 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:25.042821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:25.042947 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:25.043315 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:25.043320 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:25.043322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:25.043693 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:30:25.043959 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:25.044258 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 17:30:25.044463 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:30:25.045373 19987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:30:25.139794 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 19855
I20260923 17:30:25.159107 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 19998
Sep 23 17:30:25 dist-test-slave-p7jt krb5kdc[19160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184620, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:30:25.175173 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:30:25.159845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45208 (local address 127.8.29.195:32797)
0923 17:30:25.160028 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:30:25.160034 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:30:25.160121 (+ 87us) server_negotiation.cc:407] Connection header received
0923 17:30:25.160310 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:30:25.160314 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:30:25.160388 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:30:25.160473 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:30:25.161753 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:25.162550 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:25.163834 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:30:25.164286 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:30:25.164409 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:30:25.168710 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:30:25.168724 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:30:25.168747 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:30:25.168815 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:30:25.171762 (+ 2947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:25.172603 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:25.172615 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:25.172618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:25.172823 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:30:25.173320 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:30:25.173326 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:30:25.173330 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:30:25.173846 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:30:25.174195 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0923 17:30:25.174515 (+ 320us) server_negotiation.cc:299] Negotiation successful
0923 17:30:25.174775 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:30:25.175945 20121 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:30:25.322602 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 19998
I20260923 17:30:25.338363 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 19604
I20260923 17:30:25.339593 19666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:30:25.448458 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 19604
I20260923 17:30:25.466696 8311 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:30:25.802785 8311 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:30:25.803018 8311 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:30:26.139008 8311 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:30:26.139 UTC [19190] LOG: received smart shutdown request
2026-09-23 17:30:26.143 UTC [19190] LOG: background worker "logical replication launcher" (PID 19199) exited with exit code 1
2026-09-23 17:30:26.143 UTC [19194] LOG: shutting down
2026-09-23 17:30:26.331 UTC [19190] LOG: database system is shut down
2026-09-23T17:30:26Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51606 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:30:26 dist-test-slave-p7jt krb5kdc[20184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:30:26 dist-test-slave-p7jt krb5kdc[20184](info): set up 2 sockets
Sep 23 17:30:26 dist-test-slave-p7jt krb5kdc[20184](info): commencing operation
krb5kdc: starting...
W20260923 17:30:27.470695 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:30:27 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184627, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:30:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:30:27Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.29.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.29.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/rangeradmin_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/rangeradmin_127.8.29.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.29.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.29.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/rangerlookup_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/rangerlookup_127.8.29.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.29.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.29.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/HTTP_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/HTTP_127.8.29.212@KRBTEST.COM.keytab.
I20260923 17:30:27.787591 8311 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task6Gj89p/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/postgres -l logfile start
2026-09-23 17:30:30.589 UTC [20214] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 17:30:30.589 UTC [20214] LOG: listening on IPv4 address "127.8.29.212", port 39205
2026-09-23 17:30:30.593 UTC [20214] LOG: listening on Unix socket "/tmp/.s.PGSQL.39205"
2026-09-23 17:30:30.615 UTC [20217] LOG: database system was shut down at 2026-09-23 17:30:28 UTC
2026-09-23 17:30:30.618 UTC [20214] LOG: database system is ready to accept connections
I20260923 17:30:31.506281 8311 mini_postgres.cc:96] Postgres bound to 39205
2026-09-23 17:30:31.522 UTC [20225] FATAL: database "slave" does not exist
127.8.29.212:39205 - accepting connections
I20260923 17:30:31.523512 8311 mini_ranger.cc:161] Starting Ranger...
I20260923 17:30:31.548909 8311 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 17:30:32.069029 8311 mini_ranger.cc:87] Created ranger Postgres database
I20260923 17:30:32.069142 8311 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-admin
2026-09-23 17:30:32,349 [I] DB FLAVOR :POSTGRES
2026-09-23 17:30:32,349 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:30:32,349 [I] Checking connection..
2026-09-23 17:30:32,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 17:30:32,606 [I] Checking connection passed.
2026-09-23 17:30:32,606 [I] --------- Verifying version history table ---------
2026-09-23 17:30:32,606 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:30:32,863 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:30:32,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:30:33,131 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 17:30:33,131 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 17:30:33,131 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 17:30:33.395 UTC [20289] WARNING: there is no transaction in progress
2026-09-23 17:30:33,402 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 17:30:33,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 17:30:33,661 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 17:30:33,662 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:30:33,662 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 17:30:33,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 17:30:34,194 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-p7jt.c.gcp-upstream.internal','N') ;"
2026-09-23 17:30:34,461 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:30:34,710 [I] Table x_portal_user does not exist in database ranger
2026-09-23 17:30:34,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:30:34,965 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 17:30:34,965 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:30:35,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:30:35,481 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 17:30:35,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 17:30:36.485 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.502 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.520 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.543 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.561 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.580 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.638 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.644 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.661 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.671 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.683 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.697 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.708 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.714 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.722 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:36.729 UTC [20397] WARNING: there is no transaction in progress
2026-09-23 17:30:37,449 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 17:30:37,450 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 17:30:37,687 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 17:30:37,688 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 17:30:37,935 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 17:30:37,935 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 17:30:38,195 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 17:30:38,195 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 17:30:38,452 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 17:30:38,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-p7jt.c.gcp-upstream.internal';"
2026-09-23 17:30:38,939 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 17:30:38,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 17:30:39,179 [I] DB_PATCHES have already been applied
I20260923 17:30:39.185190 8311 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-admin:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task6Gj89p/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 17:30:39.185284 8311 mini_ranger.cc:193] Using host: 127.8.29.212
I20260923 17:30:39.199069 8311 mini_ranger.cc:227] Ranger admin URL: http://127.8.29.212:47445
Sep 23, 2026 5:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 5:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 5:30:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 5:30:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 5:30:40 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 17:30:40.239159 8311 mini_ranger.cc:160] Time spent starting Ranger: real 8.716s user 0.006s sys 0.035s
Sep 23, 2026 5:31:01 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 5:31:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 23, 2026 5:31:01 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 5:31:01 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 5:31:02 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 17:31:02.973891 8311 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.29.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.29.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/rangerkms_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.29.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/rangerkms_127.8.29.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.29.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.29.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.29.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/HTTP_127.8.29.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.29.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/HTTP_127.8.29.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 17:31:03 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184663, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 17:31:03.141431 8311 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 17:31:03.162106 8311 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 17:31:04.153005 8311 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 17:31:04.153107 8311 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-kms
I20260923 17:31:04.153139 8311 mini_ranger_kms.cc:227] Using postgres at 127.8.29.212:39205
2026-09-23 17:31:04,481 [I] DB FLAVOR :POSTGRES
2026-09-23 17:31:04,482 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 17:31:04,482 [I] Checking connection
2026-09-23 17:31:04,482 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 17:31:04,719 [I] connection success
2026-09-23 17:31:04,719 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 17:31:04,719 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 17:31:04,719 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 17:31:04,952 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 17:31:04,952 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 17:31:04,952 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 17:31:04,952 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.29.212:39205/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 17:31:05,217 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 17:31:05.670935 8311 mini_ranger_kms.cc:310] Created kms service
I20260923 17:31:05.853302 8311 mini_ranger_kms.cc:326] Created kudu user
I20260923 17:31:05.900928 8311 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 17:31:05 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184663, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:06.365548 8311 mini_ranger_kms.cc:384] Added ranger policy
I20260923 17:31:06.365867 8311 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-kms:/tmp/dist-test-task6Gj89p/build/asan/bin/postgresql.jar:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task6Gj89p/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 17:31:06.365922 8311 mini_ranger_kms.cc:240] Using host: 127.8.29.212
I20260923 17:31:06.380446 8311 mini_ranger_kms.cc:276] Ranger KMS PID: 20578
I20260923 17:31:06.380582 8311 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.29.212:54345
Sep 23, 2026 5:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 5:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 5:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 5:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 5:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 5:31:07 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 17:31:08.404788 8311 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.263s user 0.069s sys 0.280s
I20260923 17:31:08.405043 8311 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 17:31:08.405220 8311 mini_ranger_kms.cc:401] 127.8.29.212:54345/kms/v1/keys
Sep 23 17:31:08 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184668, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.29.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 17:31:09 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184668, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.29.212@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
Sep 23 17:31:09 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184668, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.29.212@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
Sep 23 17:31:11 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:31:11.759816 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39533
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:39533
--ranger_config_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:54345/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:12.082552 20628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:12.082888 20628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:12.083024 20628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:12.090387 20628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:12.090549 20628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:12.090629 20628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:12.090701 20628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:12.090761 20628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:12.101534 20628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.29.212:54345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:39533
--ranger_config_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:12.102873 20628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:12.104200 20628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:12.117266 20634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:12.117420 20633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:12.117488 20628 server_base.cc:1034] running on GCE node
W20260923 17:31:12.117712 20636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:12.118712 20628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:12.120261 20628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:12.121497 20628 hybrid_clock.cc:648] HybridClock initialized: now 1790184672121437 us; error 60 us; skew 500 ppm
I20260923 17:31:12.121848 20628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:12 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:12.125279 20628 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:31:12.127020 20628 webserver.cc:460] Webserver started at http://127.8.29.254:45197/ using document root <none> and password file <none>
I20260923 17:31:12.127770 20628 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:12.127884 20628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:12.128206 20628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:31:12 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184672, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:12.187243 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "63e2ce2f754345f385bd9175015f68c5"
format_stamp: "Formatted at 2026-09-23 17:31:12 on dist-test-slave-p7jt"
server_key: "d0c7be615147d2588df64cfa46f95d52"
server_key_iv: "24f716ca9683b79e8f2848a07b79ec1d"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:12.188138 20628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "63e2ce2f754345f385bd9175015f68c5"
format_stamp: "Formatted at 2026-09-23 17:31:12 on dist-test-slave-p7jt"
server_key: "d0c7be615147d2588df64cfa46f95d52"
server_key_iv: "24f716ca9683b79e8f2848a07b79ec1d"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:12.193739 20628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:31:12.211107 20643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:12.212775 20628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260923 17:31:12.212993 20628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "63e2ce2f754345f385bd9175015f68c5"
format_stamp: "Formatted at 2026-09-23 17:31:12 on dist-test-slave-p7jt"
server_key: "d0c7be615147d2588df64cfa46f95d52"
server_key_iv: "24f716ca9683b79e8f2848a07b79ec1d"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:12.213212 20628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:12.247991 20628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:12.252112 20628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:12.252419 20628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:12.271121 20628 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:39533
I20260923 17:31:12.271139 20694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:39533 every 8 connection(s)
I20260923 17:31:12.272544 20628 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:31:12.276350 20695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:12.276388 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 20628
I20260923 17:31:12.276563 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:31:12.277206 8311 external_mini_cluster.cc:1428] Setting key faed944b7b6df872a7dc66d06cd37778
I20260923 17:31:12.290584 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Bootstrap starting.
Sep 23 17:31:12 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:12.294138 20695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:12.295560 20695 log.cc:824] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:12.299024 20695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: No bootstrap required, opened a new log
I20260923 17:31:12.301879 20698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:12.279351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45192 (local address 127.8.29.254:39533)
0923 17:31:12.280179 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:12.280195 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:12.280249 (+ 54us) server_negotiation.cc:407] Connection header received
0923 17:31:12.282781 (+ 2532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:12.282814 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:12.283384 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:12.283869 (+ 485us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:12.285182 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:12.287165 (+ 1983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:12.288487 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:12.289531 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:12.289706 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:12.293581 (+ 3875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:12.293591 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:12.293611 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:12.293660 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:12.297183 (+ 3523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:12.297909 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:12.297917 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:12.297926 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:12.298140 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:12.298544 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:12.298548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:12.298552 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:12.299257 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:12.299546 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:12.299976 (+ 430us) server_negotiation.cc:299] Negotiation successful
0923 17:31:12.300397 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":350,"threads_started":1}
I20260923 17:31:12.307688 20695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } }
I20260923 17:31:12.308094 20695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:12.308203 20695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e2ce2f754345f385bd9175015f68c5, State: Initialized, Role: FOLLOWER
I20260923 17:31:12.308761 20695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } }
I20260923 17:31:12.308955 20695 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:31:12.309073 20695 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:31:12.309233 20695 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:12.311944 20695 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } }
I20260923 17:31:12.312510 20695 leader_election.cc:302] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e2ce2f754345f385bd9175015f68c5; no voters:
I20260923 17:31:12.313203 20695 leader_election.cc:288] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:31:12.313308 20700 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:31:12.313934 20700 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 1 LEADER]: Becoming Leader. State: Replica: 63e2ce2f754345f385bd9175015f68c5, State: Running, Role: LEADER
I20260923 17:31:12.314592 20700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } }
I20260923 17:31:12.315578 20695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:31:12.317250 20701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e2ce2f754345f385bd9175015f68c5. Latest consensus state: current_term: 1 leader_uuid: "63e2ce2f754345f385bd9175015f68c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } } }
I20260923 17:31:12.317603 20701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:12.322209 20705 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:31:12.322057 20702 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63e2ce2f754345f385bd9175015f68c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } } }
I20260923 17:31:12.322343 20702 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:12.328227 20705 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:31:12.328954 20695 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 17:31:12.348585 20705 catalog_manager.cc:1347] Generated new cluster ID: 7a860445b76d486a9e912d2dde1720ae
I20260923 17:31:12.348758 20705 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:31:12.367596 20705 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:31:12.369714 20705 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:31:12.385600 20705 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Generated new TSK 0
I20260923 17:31:12.386955 20705 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:31:13.095229 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:54345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39533
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:31:13 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184673, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 17:31:13.459292 20740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:13.459707 20740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:13.459889 20740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:13.467564 20740 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:13.467695 20740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:13.467859 20740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:31:13.479737 20740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.29.212:54345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:13.481089 20740 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:13.482604 20740 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:13.498409 20750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:13.499423 20751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:13.508612 20740 server_base.cc:1034] running on GCE node
W20260923 17:31:13.508893 20754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:13.509780 20740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:13.511116 20740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:13.512373 20740 hybrid_clock.cc:648] HybridClock initialized: now 1790184673512330 us; error 45 us; skew 500 ppm
I20260923 17:31:13.512854 20740 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:13 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184673, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:13.516908 20740 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:31:13.519735 20740 webserver.cc:460] Webserver started at http://127.8.29.193:36179/ using document root <none> and password file <none>
I20260923 17:31:13.520529 20740 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:13.520648 20740 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:13.520949 20740 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:31:13 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184673, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:13.541225 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "54a3872212f04e65bb7465af68514a5a"
format_stamp: "Formatted at 2026-09-23 17:31:13 on dist-test-slave-p7jt"
server_key: "9e3de1f88c98c4e5b6988ce5bab46b2d"
server_key_iv: "d5ce88113ddd76e83a692fe0f3c61b4f"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:13.542652 20740 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "54a3872212f04e65bb7465af68514a5a"
format_stamp: "Formatted at 2026-09-23 17:31:13 on dist-test-slave-p7jt"
server_key: "9e3de1f88c98c4e5b6988ce5bab46b2d"
server_key_iv: "d5ce88113ddd76e83a692fe0f3c61b4f"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:13.553983 20740 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.005s sys 0.004s
I20260923 17:31:13.580932 20760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:13.583139 20740 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.006s sys 0.004s
I20260923 17:31:13.583467 20740 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "54a3872212f04e65bb7465af68514a5a"
format_stamp: "Formatted at 2026-09-23 17:31:13 on dist-test-slave-p7jt"
server_key: "9e3de1f88c98c4e5b6988ce5bab46b2d"
server_key_iv: "d5ce88113ddd76e83a692fe0f3c61b4f"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:13.583873 20740 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:13.613315 20740 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:13.619366 20740 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:13.619753 20740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:13.621171 20740 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:13.623272 20740 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:13.623373 20740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:13.623590 20740 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:13.623690 20740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:13.690737 20740 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:41433
I20260923 17:31:13.690781 20872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:41433 every 8 connection(s)
I20260923 17:31:13.692058 20740 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:31:13.693542 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 20740
I20260923 17:31:13.693753 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:31:13.694635 8311 external_mini_cluster.cc:1428] Setting key b417cbd2a6b2eecf9cb2a6cf909e4107
Sep 23 17:31:13 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184673, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:13.729624 20875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:13.695893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39533 (local address 127.8.29.193:49881)
0923 17:31:13.706123 (+ 10230us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:13.706188 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:13.709526 (+ 3338us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:13.710509 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:13.710523 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:13.711206 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:13.712198 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:13.712233 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:13.713702 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:13.713708 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:13.715615 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:13.715630 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:13.716758 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:13.716766 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:13.717082 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:13.717889 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:13.717913 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:13.720385 (+ 2472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:13.724493 (+ 4108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:13.724504 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:13.724525 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:13.725170 (+ 645us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:13.725648 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:13.725653 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:13.725658 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:13.725944 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:13.726701 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:13.726710 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:13.727230 (+ 520us) client_negotiation.cc:769] Sending connection context
0923 17:31:13.727571 (+ 341us) client_negotiation.cc:240] Negotiation successful
0923 17:31:13.728160 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9846,"thread_start_us":10717,"threads_started":1}
I20260923 17:31:13.730840 20876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:13.696812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:49881 (local address 127.8.29.254:39533)
0923 17:31:13.708641 (+ 11829us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:13.708647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:13.708670 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:31:13.709924 (+ 1254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:13.709930 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:13.710020 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:13.710134 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:13.712437 (+ 2303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:13.713506 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:13.715771 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:13.716518 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:13.716693 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:13.720710 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:13.720718 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:13.720722 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:13.720776 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:13.724285 (+ 3509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:13.725329 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:13.725335 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:13.725338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:13.725506 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:13.726084 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:13.726088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:13.726091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:13.726447 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:13.726623 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:13.730192 (+ 3569us) server_negotiation.cc:299] Negotiation successful
0923 17:31:13.730441 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11462,"thread_start_us":11344,"threads_started":1}
I20260923 17:31:13.733183 20873 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39533
I20260923 17:31:13.733510 20873 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:13.734417 20873 heartbeater.cc:511] Master 127.8.29.254:39533 requested a full tablet report, sending...
I20260923 17:31:13.737116 20660 ts_manager.cc:194] Registered new tserver with Master: 54a3872212f04e65bb7465af68514a5a (127.8.29.193:41433)
I20260923 17:31:13.739246 20660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:49881
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:31:13.783088 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:54345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39533
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:31:13 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184673, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
Sep 23 17:31:14 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184673, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
W20260923 17:31:14.259800 20879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:14.260129 20879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:14.260277 20879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:14.270520 20879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:14.270761 20879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:14.271013 20879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:31:14.285970 20879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.29.212:54345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:14.288009 20879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:14.289881 20879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:14.343180 20887 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:14.348264 20886 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:14.348738 20889 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:14.356180 20879 server_base.cc:1034] running on GCE node
I20260923 17:31:14.357319 20879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:14.359488 20879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:14.360819 20879 hybrid_clock.cc:648] HybridClock initialized: now 1790184674360732 us; error 83 us; skew 500 ppm
I20260923 17:31:14.361292 20879 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:14 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184674, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:14.366051 20879 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:31:14.388175 20879 webserver.cc:460] Webserver started at http://127.8.29.194:44319/ using document root <none> and password file <none>
I20260923 17:31:14.389515 20879 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:14.389889 20879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:14.390563 20879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:31:14 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184674, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:14.423882 20879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "ce38773927974c6ebca132ed52025789"
format_stamp: "Formatted at 2026-09-23 17:31:14 on dist-test-slave-p7jt"
server_key: "f1f31d7e1d21f06e50dda256083302b6"
server_key_iv: "15c891c0e7b7641035aa1a82f68b2446"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:14.427516 20879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "ce38773927974c6ebca132ed52025789"
format_stamp: "Formatted at 2026-09-23 17:31:14 on dist-test-slave-p7jt"
server_key: "f1f31d7e1d21f06e50dda256083302b6"
server_key_iv: "15c891c0e7b7641035aa1a82f68b2446"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:14.441500 20879 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.008s sys 0.004s
I20260923 17:31:14.480715 20898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:14.482931 20879 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.006s sys 0.004s
I20260923 17:31:14.483219 20879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "ce38773927974c6ebca132ed52025789"
format_stamp: "Formatted at 2026-09-23 17:31:14 on dist-test-slave-p7jt"
server_key: "f1f31d7e1d21f06e50dda256083302b6"
server_key_iv: "15c891c0e7b7641035aa1a82f68b2446"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:14.483597 20879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:14.543015 20879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:14.548321 20879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:14.548648 20879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:14.549966 20879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:14.551816 20879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:14.551910 20879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:14.552026 20879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:14.552081 20879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:14.584039 20879 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:35493
I20260923 17:31:14.584049 21015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:35493 every 8 connection(s)
I20260923 17:31:14.585639 20879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:31:14.587922 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 20879
I20260923 17:31:14.588091 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:31:14.588557 8311 external_mini_cluster.cc:1428] Setting key dbd93754370bda447af7887c2219289c
Sep 23 17:31:14 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184674, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
I20260923 17:31:14.613457 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:14.590085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:40417 (local address 127.8.29.254:39533)
0923 17:31:14.594925 (+ 4840us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:14.594931 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:14.594952 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:31:14.595018 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:14.595023 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:14.595103 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:14.595221 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:14.597318 (+ 2097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:14.598280 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:14.600343 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:14.601007 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:14.601147 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:14.605711 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:14.605718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:14.605723 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:14.605765 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:14.609212 (+ 3447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:14.610291 (+ 1079us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:14.610297 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:14.610301 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:14.610539 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:14.611183 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:14.611188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:14.611192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:14.611599 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:14.611751 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:14.612812 (+ 1061us) server_negotiation.cc:299] Negotiation successful
0923 17:31:14.613031 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4696,"thread_start_us":4558,"threads_started":1}
I20260923 17:31:14.614817 21018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:14.589399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39533 (local address 127.8.29.194:40417)
0923 17:31:14.590314 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:14.590366 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:14.592705 (+ 2339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:14.595573 (+ 2868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:14.595588 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:14.596243 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:14.597088 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:14.597117 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:14.598481 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:14.598487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:14.600148 (+ 1661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:14.600166 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:14.601267 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:14.601274 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:14.601577 (+ 303us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:14.602438 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:14.602515 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:14.605394 (+ 2879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:14.609426 (+ 4032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:14.609438 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:14.609458 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:14.610127 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:14.610669 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:14.610673 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:14.610676 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:14.611036 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:14.611781 (+ 745us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:14.611791 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:14.612335 (+ 544us) client_negotiation.cc:769] Sending connection context
0923 17:31:14.612713 (+ 378us) client_negotiation.cc:240] Negotiation successful
0923 17:31:14.613337 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":725,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
I20260923 17:31:14.616550 21016 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39533
I20260923 17:31:14.616932 21016 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:14.617791 21016 heartbeater.cc:511] Master 127.8.29.254:39533 requested a full tablet report, sending...
I20260923 17:31:14.619568 20660 ts_manager.cc:194] Registered new tserver with Master: ce38773927974c6ebca132ed52025789 (127.8.29.194:35493)
I20260923 17:31:14.620647 20660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:40417
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:31:14.642783 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:54345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39533
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:31:14.743299 20873 heartbeater.cc:503] Master 127.8.29.254:39533 was elected leader, sending a full tablet report...
W20260923 17:31:14.973845 21022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:14.974152 21022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:14.974285 21022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:14.981709 21022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:14.981848 21022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:14.982002 21022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:31:14.992274 21022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.29.212:54345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:14.993654 21022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:14.995041 21022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:15.008946 21028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:15.008947 21027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:15.009086 21030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:15.010061 21022 server_base.cc:1034] running on GCE node
I20260923 17:31:15.010759 21022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:15.011755 21022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:15.012970 21022 hybrid_clock.cc:648] HybridClock initialized: now 1790184675012914 us; error 48 us; skew 500 ppm
I20260923 17:31:15.013283 21022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:15 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184675, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:15.016582 21022 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:31:15.018303 21022 webserver.cc:460] Webserver started at http://127.8.29.195:36247/ using document root <none> and password file <none>
I20260923 17:31:15.019052 21022 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:15.019165 21022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:15.019460 21022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 17:31:15 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184675, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:15.037483 21022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "29e8beb78f4240babd8cf9da33d01078"
format_stamp: "Formatted at 2026-09-23 17:31:15 on dist-test-slave-p7jt"
server_key: "fddd54a0c79e3e2052f8205fefaad642"
server_key_iv: "b56f2ac9b81a060458581fa1204cf4d7"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:15.038661 21022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "29e8beb78f4240babd8cf9da33d01078"
format_stamp: "Formatted at 2026-09-23 17:31:15 on dist-test-slave-p7jt"
server_key: "fddd54a0c79e3e2052f8205fefaad642"
server_key_iv: "b56f2ac9b81a060458581fa1204cf4d7"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:15.046114 21022 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260923 17:31:15.066861 21037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:15.068984 21022 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.005s
I20260923 17:31:15.069208 21022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "29e8beb78f4240babd8cf9da33d01078"
format_stamp: "Formatted at 2026-09-23 17:31:15 on dist-test-slave-p7jt"
server_key: "fddd54a0c79e3e2052f8205fefaad642"
server_key_iv: "b56f2ac9b81a060458581fa1204cf4d7"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:15.069487 21022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:15.095784 21022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:15.100010 21022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:15.100293 21022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:15.101279 21022 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:15.102677 21022 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:15.102751 21022 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:15.102838 21022 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:15.102880 21022 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:15.133273 21022 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:36525
I20260923 17:31:15.133309 21149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:36525 every 8 connection(s)
I20260923 17:31:15.134557 21022 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:31:15.138957 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 21022
I20260923 17:31:15.139122 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:31:15.139547 8311 external_mini_cluster.cc:1428] Setting key d7f77e8aedb4140a78d20a75c580fc68
Sep 23 17:31:15 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184675, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:15.154388 21152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:15.137111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:42639 (local address 127.8.29.254:39533)
0923 17:31:15.137614 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:15.137621 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:15.138901 (+ 1280us) server_negotiation.cc:407] Connection header received
0923 17:31:15.140425 (+ 1524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:15.140431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:15.140503 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:15.140592 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:15.142323 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.143255 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.144801 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.145217 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.145317 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:15.148578 (+ 3261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:15.148584 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:15.148587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:15.148622 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:15.151239 (+ 2617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:15.152046 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:15.152049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:15.152052 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:15.152177 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:15.152610 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:15.152613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:15.152615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:15.152885 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:15.153033 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:15.153818 (+ 785us) server_negotiation.cc:299] Negotiation successful
0923 17:31:15.154033 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":204,"threads_started":1}
I20260923 17:31:15.155654 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:15.137683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39533 (local address 127.8.29.195:42639)
0923 17:31:15.138413 (+ 730us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:15.138486 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:15.140170 (+ 1684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:15.140887 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:15.140900 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:15.141338 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:15.142142 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:15.142169 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.143405 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.143409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:15.144663 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:15.144676 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.145426 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.145431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:15.145644 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:15.146284 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:15.146306 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:15.148345 (+ 2039us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:15.151407 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:15.151423 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:15.151438 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:15.151944 (+ 506us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:15.152284 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:15.152287 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:15.152289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:15.152524 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:15.153030 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:15.153036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:15.153418 (+ 382us) client_negotiation.cc:769] Sending connection context
0923 17:31:15.153724 (+ 306us) client_negotiation.cc:240] Negotiation successful
0923 17:31:15.154264 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":299,"threads_started":1}
I20260923 17:31:15.157159 21150 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39533
I20260923 17:31:15.157462 21150 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:15.158183 21150 heartbeater.cc:511] Master 127.8.29.254:39533 requested a full tablet report, sending...
I20260923 17:31:15.159713 20660 ts_manager.cc:194] Registered new tserver with Master: 29e8beb78f4240babd8cf9da33d01078 (127.8.29.195:36525)
I20260923 17:31:15.160547 20660 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:42639
I20260923 17:31:15.168429 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:31:15.170231 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 20740
Sep 23 17:31:15 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:31:15.188323 21155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:15.171448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57704 (local address 127.8.29.193:41433)
0923 17:31:15.172075 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:15.172084 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:15.172113 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:31:15.172177 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:15.172183 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:15.172413 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:15.172582 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:15.174004 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.175251 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.177006 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.177634 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.177801 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:15.182330 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:15.182346 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:15.182365 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:15.182408 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:15.185122 (+ 2714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:15.185985 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:15.185993 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:15.185995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:15.186165 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:15.186679 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:15.186683 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:15.186685 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:15.187113 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:15.187375 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:15.187794 (+ 419us) server_negotiation.cc:299] Negotiation successful
0923 17:31:15.188019 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":316,"threads_started":1}
I20260923 17:31:15.189729 20868 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:15.274986 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 20740
I20260923 17:31:15.289919 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 20879
Sep 23 17:31:15 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:31:15.305691 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:15.290867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35390 (local address 127.8.29.194:35493)
0923 17:31:15.291396 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:15.291402 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:15.291427 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:31:15.291488 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:15.291494 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:15.291704 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:15.291841 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:15.293185 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.294291 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.295708 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.296280 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.296383 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:15.300345 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:15.300359 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:15.300375 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:15.300412 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:15.303208 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:15.303843 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:15.303850 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:15.303852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:15.304012 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:15.304293 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:15.304295 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:15.304297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:15.304632 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:15.304854 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:15.305144 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 17:31:15.305340 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":268,"threads_started":1}
I20260923 17:31:15.307029 21011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:15.387434 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 20879
I20260923 17:31:15.400019 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 21022
Sep 23 17:31:15 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184671, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:31:15.417219 21159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:15.400975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.8.29.195:36525)
0923 17:31:15.401605 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:15.401612 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:15.401640 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:31:15.401698 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:15.401705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:15.401926 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:15.402053 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:15.403519 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.404624 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.406017 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:15.406673 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:15.406775 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:15.411306 (+ 4531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:15.411321 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:15.411338 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:15.411385 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:15.414036 (+ 2651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:15.414937 (+ 901us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:15.414946 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:15.414948 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:15.415104 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:15.415593 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:15.415599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:15.415602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:15.415994 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:15.416235 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:15.416565 (+ 330us) server_negotiation.cc:299] Negotiation successful
0923 17:31:15.416896 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":301,"threads_started":1}
I20260923 17:31:15.418718 21145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:15.500973 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 21022
I20260923 17:31:15.513666 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 20628
I20260923 17:31:15.515164 20690 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:15.622223 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 20628
I20260923 17:31:15.639875 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39533
--webserver_interface=127.8.29.254
--webserver_port=45197
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:39533
--ranger_config_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:54345/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.29.212:54345/kms with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:15.958354 21162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:15.958709 21162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:15.958840 21162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:15.966162 21162 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:15.966285 21162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:15.966332 21162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:15.966413 21162 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:15.966492 21162 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:15.976656 21162 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.29.212:54345/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:39533
--ranger_config_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=45197
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:15.977947 21162 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:15.979254 21162 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:15.991146 21167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:15.991181 21168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:15.991238 21170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:15.992178 21162 server_base.cc:1034] running on GCE node
I20260923 17:31:15.992986 21162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:15.994504 21162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:15.995750 21162 hybrid_clock.cc:648] HybridClock initialized: now 1790184675995690 us; error 58 us; skew 500 ppm
I20260923 17:31:15.996109 21162 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:15 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184675, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:15.999141 21162 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:31:16.000800 21162 webserver.cc:460] Webserver started at http://127.8.29.254:45197/ using document root <none> and password file <none>
I20260923 17:31:16.001485 21162 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:16.001597 21162 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:31:16 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184675, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:16.087301 21162 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 17:31:16.090405 21178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:16.092134 21162 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:31:16.092324 21162 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "63e2ce2f754345f385bd9175015f68c5"
format_stamp: "Formatted at 2026-09-23 17:31:12 on dist-test-slave-p7jt"
server_key: "d0c7be615147d2588df64cfa46f95d52"
server_key_iv: "24f716ca9683b79e8f2848a07b79ec1d"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:16.093339 21162 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:16.119315 21162 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:16.123412 21162 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:16.123682 21162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:16.140748 21162 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:39533
I20260923 17:31:16.140769 21229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:39533 every 8 connection(s)
I20260923 17:31:16.142217 21162 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:31:16.143326 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 21162
I20260923 17:31:16.144137 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:54345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:41433
--local_ip_for_outbound_sockets=127.8.29.193
--tserver_master_addrs=127.8.29.254:39533
--webserver_port=36179
--webserver_interface=127.8.29.193
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.29.212:54345/kms with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:31:16.149358 21230 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:31:16.152787 21230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Bootstrap starting.
I20260923 17:31:16.171746 21230 log.cc:824] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:16.182695 21230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:31:16.183286 21230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Bootstrap complete.
I20260923 17:31:16.193003 21230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } }
I20260923 17:31:16.193861 21230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63e2ce2f754345f385bd9175015f68c5, State: Initialized, Role: FOLLOWER
I20260923 17:31:16.194590 21230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } }
I20260923 17:31:16.194816 21230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:31:16.194932 21230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:31:16.195084 21230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:31:16.199200 21230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } }
I20260923 17:31:16.199702 21230 leader_election.cc:302] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63e2ce2f754345f385bd9175015f68c5; no voters:
I20260923 17:31:16.200281 21230 leader_election.cc:288] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:31:16.200433 21234 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:31:16.201526 21234 raft_consensus.cc:686] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [term 2 LEADER]: Becoming Leader. State: Replica: 63e2ce2f754345f385bd9175015f68c5, State: Running, Role: LEADER
I20260923 17:31:16.202142 21234 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } }
I20260923 17:31:16.202430 21230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:31:16.204661 21235 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "63e2ce2f754345f385bd9175015f68c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } } }
I20260923 17:31:16.205087 21235 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:16.206720 21239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:31:16.206552 21236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63e2ce2f754345f385bd9175015f68c5. Latest consensus state: current_term: 2 leader_uuid: "63e2ce2f754345f385bd9175015f68c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63e2ce2f754345f385bd9175015f68c5" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39533 } } }
I20260923 17:31:16.206892 21236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:16.212994 21239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:31:16.214527 21239 catalog_manager.cc:1259] Loaded cluster ID: 7a860445b76d486a9e912d2dde1720ae
I20260923 17:31:16.214707 21239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:31:16.218743 21239 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:31:16.220366 21239 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Loaded TSK: 0
I20260923 17:31:16.222313 21239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:31:16.225648 21230 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260923 17:31:16.589228 21232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:16.590133 21232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:16.590337 21232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:16.602020 21232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:16.602253 21232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:16.602511 21232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:31:16.618106 21232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.29.212:54345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:41433
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=36179
--tserver_master_addrs=127.8.29.254:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:16.620220 21232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:16.622174 21232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:16.651397 21267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:16.660095 21268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:16.668028 21270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:16.668663 21232 server_base.cc:1034] running on GCE node
I20260923 17:31:16.669768 21232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:16.671185 21232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:16.672513 21232 hybrid_clock.cc:648] HybridClock initialized: now 1790184676672433 us; error 62 us; skew 500 ppm
I20260923 17:31:16.672928 21232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:16 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184676, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:16.677445 21232 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:31:16.680035 21232 webserver.cc:460] Webserver started at http://127.8.29.193:36179/ using document root <none> and password file <none>
I20260923 17:31:16.681041 21232 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:16.681272 21232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:31:16 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184676, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:16.743012 21232 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.004s sys 0.002s
I20260923 17:31:16.759236 21278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:16.762020 21232 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.006s sys 0.011s
I20260923 17:31:16.762300 21232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "54a3872212f04e65bb7465af68514a5a"
format_stamp: "Formatted at 2026-09-23 17:31:13 on dist-test-slave-p7jt"
server_key: "9e3de1f88c98c4e5b6988ce5bab46b2d"
server_key_iv: "d5ce88113ddd76e83a692fe0f3c61b4f"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:16.763800 21232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:16.807430 21232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:16.812695 21232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:16.813156 21232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:16.817632 21232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:16.820398 21232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:16.820511 21232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:16.820624 21232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:16.820698 21232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:16.907582 21232 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:41433
I20260923 17:31:16.907636 21391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:41433 every 8 connection(s)
I20260923 17:31:16.909400 21232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:31:16.914718 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 21232
I20260923 17:31:16.915570 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:54345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:35493
--local_ip_for_outbound_sockets=127.8.29.194
--tserver_master_addrs=127.8.29.254:39533
--webserver_port=44319
--webserver_interface=127.8.29.194
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.29.212:54345/kms with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:31:16 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184676, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:16.944769 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:16.913316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39533 (local address 127.8.29.193:58731)
0923 17:31:16.919223 (+ 5907us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:16.919286 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:16.921284 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:16.923150 (+ 1866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:16.923164 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:16.923701 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:16.926068 (+ 2367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:16.926094 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:16.928542 (+ 2448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:16.928548 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:16.930071 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:16.930086 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:16.931256 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:16.931266 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:16.931544 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:16.932214 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:16.932244 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:16.935019 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:16.938724 (+ 3705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:16.938735 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:16.938753 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:16.939407 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:16.940018 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:16.940024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:16.940028 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:16.940318 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:16.941647 (+ 1329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:16.941657 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:16.942153 (+ 496us) client_negotiation.cc:769] Sending connection context
0923 17:31:16.942584 (+ 431us) client_negotiation.cc:240] Negotiation successful
0923 17:31:16.943218 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5570,"thread_start_us":5286,"threads_started":1}
I20260923 17:31:16.947518 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:16.919416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:58731 (local address 127.8.29.254:39533)
0923 17:31:16.920179 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:16.920201 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:16.920256 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:31:16.922015 (+ 1759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:16.922037 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:16.922559 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:16.922945 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:16.926306 (+ 3361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:16.928319 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:16.930245 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:16.930993 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:16.931400 (+ 407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:16.935407 (+ 4007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:16.935421 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:16.935446 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:16.935510 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:16.938504 (+ 2994us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:16.939615 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:16.939625 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:16.939634 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:16.939867 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:16.940485 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:16.940490 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:16.940493 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:16.941219 (+ 726us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:16.941529 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:16.945551 (+ 4022us) server_negotiation.cc:299] Negotiation successful
0923 17:31:16.946046 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":279,"threads_started":1}
I20260923 17:31:16.952301 21392 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39533
I20260923 17:31:16.952680 21392 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:16.953588 21392 heartbeater.cc:511] Master 127.8.29.254:39533 requested a full tablet report, sending...
W20260923 17:31:16.957460 21392 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:39533 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:31:16.957707 21392 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:16.958024 21392 heartbeater.cc:511] Master 127.8.29.254:39533 requested a full tablet report, sending...
I20260923 17:31:16.959591 21392 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:16.959988 21392 heartbeater.cc:511] Master 127.8.29.254:39533 requested a full tablet report, sending...
W20260923 17:31:16.961294 21392 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:31:17.321170 21396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:17.321646 21396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:17.321915 21396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:17.333616 21396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:17.333839 21396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:17.334069 21396 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:31:17.349951 21396 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.29.212:54345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:35493
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=44319
--tserver_master_addrs=127.8.29.254:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:17.351953 21396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:17.353813 21396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:17.377568 21412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:17.379750 21413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:17.387308 21415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:17.387676 21396 server_base.cc:1034] running on GCE node
I20260923 17:31:17.388628 21396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:17.389925 21396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:17.391232 21396 hybrid_clock.cc:648] HybridClock initialized: now 1790184677391152 us; error 72 us; skew 500 ppm
I20260923 17:31:17.391618 21396 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:17 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184677, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:17.396401 21396 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:31:17.398582 21396 webserver.cc:460] Webserver started at http://127.8.29.194:44319/ using document root <none> and password file <none>
I20260923 17:31:17.399552 21396 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:17.399714 21396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:31:17 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184677, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
Sep 23 17:31:17 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790184677, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:17.438580 21396 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.008s sys 0.002s
I20260923 17:31:17.454025 21423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:17.457461 21396 fs_manager.cc:725] Time spent opening block manager: real 0.017s user 0.003s sys 0.012s
I20260923 17:31:17.457681 21396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "ce38773927974c6ebca132ed52025789"
format_stamp: "Formatted at 2026-09-23 17:31:14 on dist-test-slave-p7jt"
server_key: "f1f31d7e1d21f06e50dda256083302b6"
server_key_iv: "15c891c0e7b7641035aa1a82f68b2446"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:17.458807 21396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:17.478345 21396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:17.482798 21396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:17.483070 21396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:17.484102 21396 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:17.485534 21396 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:17.485627 21396 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:17.485741 21396 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:17.485828 21396 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:17.532624 21396 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:35493
I20260923 17:31:17.532691 21536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:35493 every 8 connection(s)
I20260923 17:31:17.534953 21396 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:31:17.537088 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 21396
I20260923 17:31:17.537850 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.29.212:54345/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:36525
--local_ip_for_outbound_sockets=127.8.29.195
--tserver_master_addrs=127.8.29.254:39533
--webserver_port=36247
--webserver_interface=127.8.29.195
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:45679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.29.212:54345/kms with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:31:17 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184677, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:17.572059 21539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:17.545432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:32911 (local address 127.8.29.254:39533)
0923 17:31:17.547062 (+ 1630us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:17.547070 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:17.550514 (+ 3444us) server_negotiation.cc:407] Connection header received
0923 17:31:17.552581 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:17.552589 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:17.552688 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:17.552796 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:17.555230 (+ 2434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:17.556397 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:17.559027 (+ 2630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:17.559718 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:17.559840 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:17.564369 (+ 4529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:17.564377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:17.564382 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:17.564425 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:17.567962 (+ 3537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:17.569046 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:17.569053 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:17.569056 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:17.569205 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:17.569743 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:17.569756 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:17.569759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:17.570210 (+ 451us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:17.570367 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:17.571447 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0923 17:31:17.571681 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1486,"thread_start_us":1373,"threads_started":1}
I20260923 17:31:17.574730 21538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:17.544772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39533 (local address 127.8.29.194:32911)
0923 17:31:17.548125 (+ 3353us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:17.548188 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:17.552170 (+ 3982us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:17.553433 (+ 1263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:17.553447 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:17.554088 (+ 641us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:17.555008 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:17.555038 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:17.556616 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:17.556622 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:17.558838 (+ 2216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:17.558854 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:17.560250 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:17.560261 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:17.560591 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:17.561427 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:17.561458 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:17.564070 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:17.568137 (+ 4067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:17.568147 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:17.568168 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:17.568785 (+ 617us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:17.569329 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:17.569334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:17.569338 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:17.569586 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:17.570481 (+ 895us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:17.570490 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:17.570969 (+ 479us) client_negotiation.cc:769] Sending connection context
0923 17:31:17.572622 (+ 1653us) client_negotiation.cc:240] Negotiation successful
0923 17:31:17.573250 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3022,"thread_start_us":2717,"threads_started":1}
I20260923 17:31:17.576798 21537 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39533
I20260923 17:31:17.577153 21537 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:17.578109 21537 heartbeater.cc:511] Master 127.8.29.254:39533 requested a full tablet report, sending...
I20260923 17:31:17.580480 21195 ts_manager.cc:194] Registered new tserver with Master: ce38773927974c6ebca132ed52025789 (127.8.29.194:35493)
I20260923 17:31:17.582880 21195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:32911
Sep 23 17:31:17 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184677, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:17.961444 21392 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:17.961891 21392 heartbeater.cc:511] Master 127.8.29.254:39533 requested a full tablet report, sending...
I20260923 17:31:17.963222 21195 ts_manager.cc:194] Registered new tserver with Master: 54a3872212f04e65bb7465af68514a5a (127.8.29.193:41433)
I20260923 17:31:17.964396 21195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:58731
W20260923 17:31:17.999842 21540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:18.000399 21540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:18.000612 21540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:18.011332 21540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:18.011560 21540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:18.011790 21540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:31:18.026568 21540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.29.212:54345/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:36525
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=36247
--tserver_master_addrs=127.8.29.254:39533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:18.028473 21540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:18.030299 21540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:18.050243 21546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:18.050254 21547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:18.060443 21540 server_base.cc:1034] running on GCE node
W20260923 17:31:18.060813 21549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:18.061484 21540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:18.062548 21540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:18.063823 21540 hybrid_clock.cc:648] HybridClock initialized: now 1790184678063743 us; error 59 us; skew 500 ppm
I20260923 17:31:18.064213 21540 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:18 dist-test-slave-p7jt krb5kdc[20184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184678, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:18.072181 21540 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:31:18.075933 21540 webserver.cc:460] Webserver started at http://127.8.29.195:36247/ using document root <none> and password file <none>
I20260923 17:31:18.076678 21540 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:18.076812 21540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 17:31:18 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184678, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:18.116394 21540 fs_manager.cc:709] Time spent opening directory manager: real 0.015s user 0.004s sys 0.008s
I20260923 17:31:18.120025 21557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:18.121572 21540 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:31:18.121743 21540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "29e8beb78f4240babd8cf9da33d01078"
format_stamp: "Formatted at 2026-09-23 17:31:15 on dist-test-slave-p7jt"
server_key: "fddd54a0c79e3e2052f8205fefaad642"
server_key_iv: "b56f2ac9b81a060458581fa1204cf4d7"
server_key_version: "kuduclusterkey@0"
I20260923 17:31:18.122822 21540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:18.139953 21540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:18.145040 21540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:18.145403 21540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:18.150882 21540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:18.152508 21540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:18.152611 21540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:18.152745 21540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:18.152815 21540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:18.227221 21540 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:36525
I20260923 17:31:18.227311 21670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:36525 every 8 connection(s)
Sep 23 17:31:18 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184677, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for HTTP/127.8.29.212@KRBTEST.COM
I20260923 17:31:18.229189 21540 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:31:18.233255 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 21540
Sep 23 17:31:18 dist-test-slave-p7jt krb5kdc[20184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184678, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:18.296465 21675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.241457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:44703 (local address 127.8.29.254:39533)
0923 17:31:18.251281 (+ 9824us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.251289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.266611 (+ 15322us) server_negotiation.cc:407] Connection header received
0923 17:31:18.271239 (+ 4628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.271245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.271337 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.271455 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:18.276781 (+ 5326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.278249 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.283254 (+ 5005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.283938 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.284068 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.288613 (+ 4545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:18.288620 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:18.288625 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:18.288682 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:18.292256 (+ 3574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.293396 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.293401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.293405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.293548 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.294127 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.294132 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.294136 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.294562 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:18.294693 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.295759 (+ 1066us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.296087 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":9652,"thread_start_us":9452,"threads_started":1}
I20260923 17:31:18.299800 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.242152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39533 (local address 127.8.29.195:44703)
0923 17:31:18.266380 (+ 24228us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:18.266450 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:18.270888 (+ 4438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:18.273252 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:18.273266 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:18.273892 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:18.274777 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.274806 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.278706 (+ 3900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.278712 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.283049 (+ 4337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.283071 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.284521 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.284529 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.284850 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.285588 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:18.285632 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:18.288315 (+ 2683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:18.292432 (+ 4117us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:18.292442 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:18.292482 (+ 40us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:18.293250 (+ 768us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:18.293701 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:18.293706 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:18.293710 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:18.293995 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:18.294831 (+ 836us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:18.294840 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:18.295323 (+ 483us) client_negotiation.cc:769] Sending connection context
0923 17:31:18.297061 (+ 1738us) client_negotiation.cc:240] Negotiation successful
0923 17:31:18.297652 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22621,"thread_start_us":22670,"threads_started":1}
I20260923 17:31:18.302311 21671 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39533
I20260923 17:31:18.302683 21671 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:18.303597 21671 heartbeater.cc:511] Master 127.8.29.254:39533 requested a full tablet report, sending...
I20260923 17:31:18.305397 21195 ts_manager.cc:194] Registered new tserver with Master: 29e8beb78f4240babd8cf9da33d01078 (127.8.29.195:36525)
I20260923 17:31:18.306535 21195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:44703
I20260923 17:31:18.307065 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.234712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48754 (local address 127.8.29.254:39533)
0923 17:31:18.242836 (+ 8124us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.242843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.269266 (+ 26423us) server_negotiation.cc:407] Connection header received
0923 17:31:18.269476 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.269482 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.269577 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.269700 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:18.272901 (+ 3201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.273930 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.276696 (+ 2766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.277488 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.277616 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.287051 (+ 9435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:18.287059 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:18.287064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:18.287108 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:18.290833 (+ 3725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.299360 (+ 8527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.299366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.299369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.299513 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.303803 (+ 4290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.303809 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.303813 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.304176 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:18.304422 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.305085 (+ 663us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.305333 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":7994,"thread_start_us":7864,"threads_started":1}
I20260923 17:31:18.312325 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:31:18.413779 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.383272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48762 (local address 127.8.29.254:39533)
0923 17:31:18.383583 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.383588 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.395560 (+ 11972us) server_negotiation.cc:407] Connection header received
0923 17:31:18.395936 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.395943 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.396079 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.396177 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:18.398996 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.400119 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.402183 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.402884 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.403002 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.406699 (+ 3697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:18.406707 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:18.406712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:18.406761 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:18.410255 (+ 3494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.411259 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.411265 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.411270 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.411426 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.411926 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.411931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.411934 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.412308 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:18.412560 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.413204 (+ 644us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.413452 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:31:18.421439 21195 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:31:18.433331 21195 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:31:18.488099 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.462222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:35493 (local address 127.0.0.1:45806)
0923 17:31:18.469188 (+ 6966us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:18.469242 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:18.469430 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:18.475507 (+ 6077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:18.475513 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:18.475543 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:18.475908 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.475923 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.482489 (+ 6566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.482497 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.484660 (+ 2163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.484681 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.487044 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.487052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.487283 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.487359 (+ 76us) client_negotiation.cc:769] Sending connection context
0923 17:31:18.487515 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 17:31:18.487718 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6670,"thread_start_us":8095,"threads_started":1}
I20260923 17:31:18.490183 21686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.464230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:36525 (local address 127.0.0.1:45318)
0923 17:31:18.470584 (+ 6354us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:18.470603 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:18.470721 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:18.471783 (+ 1062us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:18.471791 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:18.471823 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:18.472207 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.472219 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.476077 (+ 3858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.476082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.478609 (+ 2527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.478629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.489357 (+ 10728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.489364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.489570 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.489606 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:31:18.489700 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:31:18.489853 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6167,"thread_start_us":5412,"threads_started":1}
I20260923 17:31:18.491583 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.464895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.8.29.195:36525)
0923 17:31:18.470123 (+ 5228us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.470132 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.470846 (+ 714us) server_negotiation.cc:407] Connection header received
0923 17:31:18.470956 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.470964 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.471247 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.471435 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:18.472425 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.474123 (+ 1698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.480616 (+ 6493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.482247 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.488910 (+ 6663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.489186 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.491029 (+ 1843us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.491230 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5046,"thread_start_us":5674,"threads_started":1}
I20260923 17:31:18.496397 21605 tablet_service.cc:1467] Processing CreateTablet for tablet 06c2e7223da34f24bc0b0e2edc1336ae (DEFAULT_TABLE table=test-table [id=f8f695c895f540e4b4916b8a22b3c072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:18.497982 21605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06c2e7223da34f24bc0b0e2edc1336ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:18.504570 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.462404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45806 (local address 127.8.29.194:35493)
0923 17:31:18.474637 (+ 12233us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.474645 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.474674 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:31:18.474774 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.474782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.475052 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.475253 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:18.478832 (+ 3579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.480403 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.484922 (+ 4519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.486502 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.486642 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.486867 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.504023 (+ 17156us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.504241 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12020,"spinlock_wait_cycles":23936,"thread_start_us":12024,"threads_started":1}
I20260923 17:31:18.511888 21472 tablet_service.cc:1467] Processing CreateTablet for tablet 06c2e7223da34f24bc0b0e2edc1336ae (DEFAULT_TABLE table=test-table [id=f8f695c895f540e4b4916b8a22b3c072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:18.514114 21472 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06c2e7223da34f24bc0b0e2edc1336ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:18.527168 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.472119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36254 (local address 127.8.29.193:41433)
0923 17:31:18.494367 (+ 22248us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.494377 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.494419 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:31:18.494555 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.494563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.494840 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.495076 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:18.503296 (+ 8220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.505023 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.514816 (+ 9793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.516481 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.516649 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.516945 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.524831 (+ 7886us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.525049 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22070,"thread_start_us":23133,"threads_started":1}
I20260923 17:31:18.527583 21688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.471616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:41433 (local address 127.0.0.1:36254)
0923 17:31:18.490709 (+ 19093us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:18.490729 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:18.490843 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:18.502889 (+ 12046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:18.502893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:18.502923 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:18.503084 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.503093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.505224 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.505230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.507574 (+ 2344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.507596 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.523870 (+ 16274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.523876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.524110 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.524148 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:31:18.524265 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 17:31:18.524447 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18965,"thread_start_us":24133,"threads_started":1}
I20260923 17:31:18.532215 21327 tablet_service.cc:1467] Processing CreateTablet for tablet 06c2e7223da34f24bc0b0e2edc1336ae (DEFAULT_TABLE table=test-table [id=f8f695c895f540e4b4916b8a22b3c072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:18.533895 21327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 06c2e7223da34f24bc0b0e2edc1336ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:18.540761 21692 tablet_bootstrap.cc:492] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: Bootstrap starting.
I20260923 17:31:18.544720 21692 tablet_bootstrap.cc:654] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:18.545971 21692 log.cc:824] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:18.554390 21696 tablet_bootstrap.cc:492] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: Bootstrap starting.
I20260923 17:31:18.558290 21696 tablet_bootstrap.cc:654] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:18.559676 21692 tablet_bootstrap.cc:492] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: No bootstrap required, opened a new log
I20260923 17:31:18.559950 21692 ts_tablet_manager.cc:1397] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.010s
I20260923 17:31:18.564896 21696 log.cc:824] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:18.566509 21697 tablet_bootstrap.cc:492] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: Bootstrap starting.
I20260923 17:31:18.570314 21696 tablet_bootstrap.cc:492] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: No bootstrap required, opened a new log
I20260923 17:31:18.570402 21697 tablet_bootstrap.cc:654] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:18.570827 21696 ts_tablet_manager.cc:1397] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.004s
I20260923 17:31:18.571672 21697 log.cc:824] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:18.572067 21692 raft_consensus.cc:348] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } }
I20260923 17:31:18.572680 21692 raft_consensus.cc:374] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:18.572835 21692 raft_consensus.cc:729] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29e8beb78f4240babd8cf9da33d01078, State: Initialized, Role: FOLLOWER
I20260923 17:31:18.573577 21692 consensus_queue.cc:260] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } }
I20260923 17:31:18.578902 21697 tablet_bootstrap.cc:492] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: No bootstrap required, opened a new log
I20260923 17:31:18.579187 21697 ts_tablet_manager.cc:1397] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20260923 17:31:18.581890 21696 raft_consensus.cc:348] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } }
I20260923 17:31:18.582372 21696 raft_consensus.cc:374] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:18.583487 21696 raft_consensus.cc:729] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce38773927974c6ebca132ed52025789, State: Initialized, Role: FOLLOWER
I20260923 17:31:18.584208 21696 consensus_queue.cc:260] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } }
I20260923 17:31:18.585384 21671 heartbeater.cc:503] Master 127.8.29.254:39533 was elected leader, sending a full tablet report...
I20260923 17:31:18.586267 21692 ts_tablet_manager.cc:1428] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: Time spent starting tablet: real 0.026s user 0.020s sys 0.001s
I20260923 17:31:18.587777 21537 heartbeater.cc:503] Master 127.8.29.254:39533 was elected leader, sending a full tablet report...
I20260923 17:31:18.589704 21697 raft_consensus.cc:348] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } }
I20260923 17:31:18.590157 21697 raft_consensus.cc:374] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:18.590269 21697 raft_consensus.cc:729] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54a3872212f04e65bb7465af68514a5a, State: Initialized, Role: FOLLOWER
I20260923 17:31:18.591585 21697 consensus_queue.cc:260] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } }
I20260923 17:31:18.593045 21392 heartbeater.cc:503] Master 127.8.29.254:39533 was elected leader, sending a full tablet report...
I20260923 17:31:18.593729 21697 ts_tablet_manager.cc:1428] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260923 17:31:18.593729 21696 ts_tablet_manager.cc:1428] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: Time spent starting tablet: real 0.023s user 0.012s sys 0.007s
I20260923 17:31:18.618839 21704 raft_consensus.cc:482] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:31:18.619161 21704 raft_consensus.cc:504] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } }
I20260923 17:31:18.620916 21704 leader_election.cc:288] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce38773927974c6ebca132ed52025789 (127.8.29.194:35493), 54a3872212f04e65bb7465af68514a5a (127.8.29.193:41433)
I20260923 17:31:18.626793 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.621395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:35773 (local address 127.8.29.194:35493)
0923 17:31:18.621673 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.621679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.621700 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:31:18.621887 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.621891 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.621952 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.622023 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:18.622384 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.623423 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.625111 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.625909 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.626076 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.626152 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.626394 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.626561 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260923 17:31:18.626788 21676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.621222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:35493 (local address 127.8.29.195:35773)
0923 17:31:18.621591 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:18.621611 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:18.621737 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:18.622059 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:18.622064 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:18.622090 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:18.622256 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.622263 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.623564 (+ 1301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.623569 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.624948 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.624960 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.626067 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.626073 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.626246 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.626271 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:31:18.626350 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 17:31:18.626510 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254}
I20260923 17:31:18.627215 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.621326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:45005 (local address 127.8.29.193:41433)
0923 17:31:18.621537 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.621543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.622100 (+ 557us) server_negotiation.cc:407] Connection header received
0923 17:31:18.622254 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.622258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.622318 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.622390 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:18.622931 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.623896 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.625473 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.626306 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.626491 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.626563 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.626842 (+ 279us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.626978 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:31:18.627452 21708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.621329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:41433 (local address 127.8.29.195:45005)
0923 17:31:18.621989 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:18.622003 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:18.622118 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:18.622419 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:18.622423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:18.622444 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:18.622770 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.622779 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.624019 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.624025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.625327 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:18.625338 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.626508 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.626515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:18.626701 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.626726 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:31:18.626809 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 17:31:18.627089 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":572,"thread_start_us":321,"threads_started":1}
I20260923 17:31:18.628165 21492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c2e7223da34f24bc0b0e2edc1336ae" candidate_uuid: "29e8beb78f4240babd8cf9da33d01078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce38773927974c6ebca132ed52025789" is_pre_election: true
I20260923 17:31:18.628293 21347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c2e7223da34f24bc0b0e2edc1336ae" candidate_uuid: "29e8beb78f4240babd8cf9da33d01078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a3872212f04e65bb7465af68514a5a" is_pre_election: true
I20260923 17:31:18.628777 21347 raft_consensus.cc:2454] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29e8beb78f4240babd8cf9da33d01078 in term 0.
I20260923 17:31:18.628777 21492 raft_consensus.cc:2454] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29e8beb78f4240babd8cf9da33d01078 in term 0.
I20260923 17:31:18.629454 21560 leader_election.cc:302] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29e8beb78f4240babd8cf9da33d01078, 54a3872212f04e65bb7465af68514a5a; no voters:
I20260923 17:31:18.629889 21704 raft_consensus.cc:2781] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:31:18.630021 21704 raft_consensus.cc:482] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:31:18.630133 21704 raft_consensus.cc:3037] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:18.632402 21704 raft_consensus.cc:504] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } }
I20260923 17:31:18.632968 21704 leader_election.cc:288] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [CANDIDATE]: Term 1 election: Requested vote from peers ce38773927974c6ebca132ed52025789 (127.8.29.194:35493), 54a3872212f04e65bb7465af68514a5a (127.8.29.193:41433)
I20260923 17:31:18.633466 21347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c2e7223da34f24bc0b0e2edc1336ae" candidate_uuid: "29e8beb78f4240babd8cf9da33d01078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "54a3872212f04e65bb7465af68514a5a"
I20260923 17:31:18.633467 21492 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "06c2e7223da34f24bc0b0e2edc1336ae" candidate_uuid: "29e8beb78f4240babd8cf9da33d01078" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce38773927974c6ebca132ed52025789"
I20260923 17:31:18.633675 21492 raft_consensus.cc:3037] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:18.633675 21347 raft_consensus.cc:3037] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:18.636817 21492 raft_consensus.cc:2454] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29e8beb78f4240babd8cf9da33d01078 in term 1.
I20260923 17:31:18.636817 21347 raft_consensus.cc:2454] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29e8beb78f4240babd8cf9da33d01078 in term 1.
I20260923 17:31:18.637356 21560 leader_election.cc:302] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 29e8beb78f4240babd8cf9da33d01078, 54a3872212f04e65bb7465af68514a5a, ce38773927974c6ebca132ed52025789; no voters:
I20260923 17:31:18.637682 21704 raft_consensus.cc:2781] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:31:18.638127 21704 raft_consensus.cc:686] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 1 LEADER]: Becoming Leader. State: Replica: 29e8beb78f4240babd8cf9da33d01078, State: Running, Role: LEADER
I20260923 17:31:18.638661 21704 consensus_queue.cc:237] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } }
I20260923 17:31:18.643146 21194 catalog_manager.cc:5495] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 reported cstate change: term changed from 0 to 1, leader changed from <none> to 29e8beb78f4240babd8cf9da33d01078 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "29e8beb78f4240babd8cf9da33d01078" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29e8beb78f4240babd8cf9da33d01078" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 36525 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 } health_report { overall_health: UNKNOWN } } }
W20260923 17:31:18.665378 21393 tablet.cc:2369] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:31:18.668195 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.660513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.8.29.195:36525)
0923 17:31:18.660730 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.660736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.660755 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:31:18.660935 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.660939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.661024 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.661155 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:31:18.661544 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.662243 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.663952 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.664575 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.666540 (+ 1965us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.666772 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:31:18.667502 (+ 730us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:31:18.667650 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.667818 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.667924 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:31:18.676726 21492 raft_consensus.cc:1261] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [term 1 FOLLOWER]: Refusing update from remote peer 29e8beb78f4240babd8cf9da33d01078: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:18.676736 21347 raft_consensus.cc:1261] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [term 1 FOLLOWER]: Refusing update from remote peer 29e8beb78f4240babd8cf9da33d01078: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:18.677675 21709 consensus_queue.cc:1035] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce38773927974c6ebca132ed52025789" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 35493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:18.677992 21704 consensus_queue.cc:1035] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [LEADER]: Connected to new peer: Peer: permanent_uuid: "54a3872212f04e65bb7465af68514a5a" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 41433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:18.688798 21712 mvcc.cc:205] Tried to move back new op lower bound from 7332596443850379264 to 7332596443707863040. Current Snapshot: MvccSnapshot[applied={T|T < 7332596443850379264}]
I20260923 17:31:18.703827 21195 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:31:18.704154 21195 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:31:18.707858 21195 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 63e2ce2f754345f385bd9175015f68c5: Sending DeleteTablet for 3 replicas of tablet 06c2e7223da34f24bc0b0e2edc1336ae
I20260923 17:31:18.708988 21472 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c2e7223da34f24bc0b0e2edc1336ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:18 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:45806
I20260923 17:31:18.708997 21605 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c2e7223da34f24bc0b0e2edc1336ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:18 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:45318
I20260923 17:31:18.709403 21327 tablet_service.cc:1514] Processing DeleteTablet for tablet 06c2e7223da34f24bc0b0e2edc1336ae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:18 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:36254
I20260923 17:31:18.709841 21721 tablet_replica.cc:330] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: stopping tablet replica
I20260923 17:31:18.710263 21721 raft_consensus.cc:2229] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:31:18.711129 21721 raft_consensus.cc:2258] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:18.713100 21721 ts_tablet_manager.cc:1905] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:18.717274 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 21232
I20260923 17:31:18.720486 21721 ts_tablet_manager.cc:1918] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:18.720643 21723 tablet_replica.cc:330] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: stopping tablet replica
I20260923 17:31:18.720666 21721 log.cc:1196] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/06c2e7223da34f24bc0b0e2edc1336ae
I20260923 17:31:18.721190 21721 ts_tablet_manager.cc:1939] T 06c2e7223da34f24bc0b0e2edc1336ae P 29e8beb78f4240babd8cf9da33d01078: Deleting consensus metadata
I20260923 17:31:18.721138 21723 raft_consensus.cc:2229] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:31:18.721683 21723 raft_consensus.cc:2258] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:18.723093 21180 catalog_manager.cc:4841] TS 29e8beb78f4240babd8cf9da33d01078 (127.8.29.195:36525): tablet 06c2e7223da34f24bc0b0e2edc1336ae (table test-table [id=f8f695c895f540e4b4916b8a22b3c072]) successfully deleted
I20260923 17:31:18.723843 21722 tablet_replica.cc:330] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: stopping tablet replica
I20260923 17:31:18.724257 21722 raft_consensus.cc:2229] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:31:18.724745 21722 raft_consensus.cc:2258] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:18.727047 21722 ts_tablet_manager.cc:1905] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:18.727094 21723 ts_tablet_manager.cc:1905] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:18.734086 21722 ts_tablet_manager.cc:1918] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:18.734087 21723 ts_tablet_manager.cc:1918] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:18.734265 21722 log.cc:1196] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/06c2e7223da34f24bc0b0e2edc1336ae
I20260923 17:31:18.734578 21723 log.cc:1196] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/06c2e7223da34f24bc0b0e2edc1336ae
I20260923 17:31:18.734858 21722 ts_tablet_manager.cc:1939] T 06c2e7223da34f24bc0b0e2edc1336ae P ce38773927974c6ebca132ed52025789: Deleting consensus metadata
I20260923 17:31:18.735244 21723 ts_tablet_manager.cc:1939] T 06c2e7223da34f24bc0b0e2edc1336ae P 54a3872212f04e65bb7465af68514a5a: Deleting consensus metadata
I20260923 17:31:18.736217 21691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.718287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36266 (local address 127.8.29.193:41433)
0923 17:31:18.718509 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.718515 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.718541 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:31:18.718643 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.718649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.718752 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.718874 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:18.720747 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.721848 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.723396 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.724014 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.724143 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.727452 (+ 3309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:18.727468 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:18.727488 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:18.727533 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:18.731627 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.732754 (+ 1127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.732765 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.732769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.732958 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.733430 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.733436 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.733439 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.733957 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:18.734283 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.735553 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.735804 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:31:18.736814 21179 catalog_manager.cc:4841] TS ce38773927974c6ebca132ed52025789 (127.8.29.194:35493): tablet 06c2e7223da34f24bc0b0e2edc1336ae (table test-table [id=f8f695c895f540e4b4916b8a22b3c072]) successfully deleted
I20260923 17:31:18.736969 21387 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:18.845786 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 21232
I20260923 17:31:18.845860 21181 catalog_manager.cc:4841] TS 54a3872212f04e65bb7465af68514a5a (127.8.29.193:41433): tablet 06c2e7223da34f24bc0b0e2edc1336ae (table test-table [id=f8f695c895f540e4b4916b8a22b3c072]) successfully deleted
I20260923 17:31:18.859267 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 21396
I20260923 17:31:18.873328 21687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.860034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.8.29.194:35493)
0923 17:31:18.860204 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.860210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.860279 (+ 69us) server_negotiation.cc:407] Connection header received
0923 17:31:18.860487 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.860493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.860569 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.860655 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:18.861958 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.862753 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.864125 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.864637 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.864770 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.867393 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:18.867413 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:18.867432 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:18.867476 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:18.870260 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.871090 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.871101 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.871103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.871251 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.871666 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.871671 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.871674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.872086 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:18.872368 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.872736 (+ 368us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.872973 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:31:18.874022 21532 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:18.963325 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 21396
I20260923 17:31:18.983700 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 21540
I20260923 17:31:18.998167 21689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:18.984573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45324 (local address 127.8.29.195:36525)
0923 17:31:18.984768 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:18.984775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:18.984793 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:31:18.984852 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:18.984855 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:18.984932 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:18.985040 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:18.986395 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.987240 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.988546 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:18.988987 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:18.989106 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:18.991778 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:18.991791 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:18.991806 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:18.991881 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:18.995023 (+ 3142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.995707 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.995714 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.995717 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.995936 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:18.996423 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:18.996427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:18.996429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:18.996841 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:18.997189 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:18.997580 (+ 391us) server_negotiation.cc:299] Negotiation successful
0923 17:31:18.997779 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:31:18.999001 21666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:19.117868 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 21540
I20260923 17:31:19.132818 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 21162
I20260923 17:31:19.134138 21225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:19.244741 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 21162
I20260923 17:31:19.262126 8311 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 17:31:19.597640 8311 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 17:31:19.597831 8311 mini_ranger.cc:66] Stopping Ranger...
I20260923 17:31:19.934269 8311 mini_ranger.cc:68] Stopped Ranger
2026-09-23 17:31:19.934 UTC [20214] LOG: received smart shutdown request
2026-09-23 17:31:19.937 UTC [20214] LOG: background worker "logical replication launcher" (PID 20223) exited with exit code 1
2026-09-23 17:31:19.938 UTC [20218] LOG: shutting down
2026-09-23 17:31:20.126 UTC [20214] LOG: database system is shut down
2026-09-23T17:31:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53779 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:31:20 dist-test-slave-p7jt krb5kdc[21736](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:31:20 dist-test-slave-p7jt krb5kdc[21736](info): set up 2 sockets
Sep 23 17:31:20 dist-test-slave-p7jt krb5kdc[21736](info): commencing operation
krb5kdc: starting...
W20260923 17:31:21.243001 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.978s user 0.003s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:31:21 dist-test-slave-p7jt krb5kdc[21736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:31:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:31:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:31:21.487414 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:46545
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:46545,127.8.29.253:37273,127.8.29.252:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:21.807080 21749 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:21.807395 21749 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:21.807554 21749 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:21.814769 21749 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:21.814895 21749 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:21.814970 21749 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:21.815044 21749 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:21.815105 21749 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:21.824999 21749 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:46545,127.8.29.253:37273,127.8.29.252:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:46545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:21.826400 21749 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:21.827766 21749 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:21.839449 21754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:21.839610 21755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:21.839476 21757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:21.840579 21749 server_base.cc:1034] running on GCE node
I20260923 17:31:21.841390 21749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:21.842998 21749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:21.844220 21749 hybrid_clock.cc:648] HybridClock initialized: now 1790184681844174 us; error 35 us; skew 500 ppm
I20260923 17:31:21.844584 21749 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:21 dist-test-slave-p7jt krb5kdc[21736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184681, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:21.847733 21749 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:31:21.849269 21749 webserver.cc:460] Webserver started at http://127.8.29.254:46843/ using document root <none> and password file <none>
I20260923 17:31:21.849947 21749 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:21.850042 21749 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:21.850291 21749 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:21.853380 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "28fa207e24cf4ef1b11225f7aad6f82c"
format_stamp: "Formatted at 2026-09-23 17:31:21 on dist-test-slave-p7jt"
server_key: "68774d3e757bf58b0996af550dfaf77f"
server_key_iv: "671bb125666ecb0e096c7a16b5d13370"
server_key_version: "encryptionkey@0"
I20260923 17:31:21.854135 21749 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "28fa207e24cf4ef1b11225f7aad6f82c"
format_stamp: "Formatted at 2026-09-23 17:31:21 on dist-test-slave-p7jt"
server_key: "68774d3e757bf58b0996af550dfaf77f"
server_key_iv: "671bb125666ecb0e096c7a16b5d13370"
server_key_version: "encryptionkey@0"
I20260923 17:31:21.859449 21749 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 17:31:21.862839 21764 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:21.864275 21749 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:31:21.864441 21749 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "28fa207e24cf4ef1b11225f7aad6f82c"
format_stamp: "Formatted at 2026-09-23 17:31:21 on dist-test-slave-p7jt"
server_key: "68774d3e757bf58b0996af550dfaf77f"
server_key_iv: "671bb125666ecb0e096c7a16b5d13370"
server_key_version: "encryptionkey@0"
I20260923 17:31:21.864636 21749 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:21.896400 21749 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:21.900713 21749 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:21.900985 21749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:21.919129 21749 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:46545
I20260923 17:31:21.919152 21815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:46545 every 8 connection(s)
I20260923 17:31:21.920467 21749 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:31:21.924166 21816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:21.928910 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 21749
I20260923 17:31:21.929109 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:31:21.929735 8311 external_mini_cluster.cc:1428] Setting key 425d67145f51dfa123bc857f27d0dd55
I20260923 17:31:21.930980 21816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:31:21 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184681, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.253@KRBTEST.COM" created.
I20260923 17:31:21.954813 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:21.934110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55942 (local address 127.8.29.254:46545)
0923 17:31:21.935839 (+ 1729us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:21.935850 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:21.935900 (+ 50us) server_negotiation.cc:407] Connection header received
0923 17:31:21.937417 (+ 1517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:21.937423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:21.937673 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:21.937835 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:21.939407 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:21.940595 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:21.942591 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:21.943158 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:21.943301 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:21.947187 (+ 3886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:21.947199 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:21.947219 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:21.947258 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:21.950734 (+ 3476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:21.951450 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:21.951457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:21.951459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:21.951606 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:21.952015 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:21.952018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:21.952021 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:21.952600 (+ 579us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:21.952757 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:21.953406 (+ 649us) server_negotiation.cc:299] Negotiation successful
0923 17:31:21.953659 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1512,"spinlock_wait_cycles":38272,"thread_start_us":1315,"threads_started":1}
I20260923 17:31:21.954887 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:21.934110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46545 (local address 127.0.0.1:55942)
0923 17:31:21.934994 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:21.935056 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:21.937050 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:21.937910 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:21.937921 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:21.938683 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:21.939207 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:21.939233 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:21.940751 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:21.940756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:21.942427 (+ 1671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:21.942443 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:21.943404 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:21.943410 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:21.943566 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:21.944256 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:21.944285 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:21.946924 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:21.950899 (+ 3975us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:21.950909 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:21.950920 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:21.951308 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:21.951709 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:21.951712 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:21.951714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:21.951915 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:21.952741 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:21.952747 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:21.953003 (+ 256us) client_negotiation.cc:769] Sending connection context
0923 17:31:21.953244 (+ 241us) client_negotiation.cc:240] Negotiation successful
0923 17:31:21.953853 (+ 609us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":95,"thread_start_us":399,"threads_started":1}
I20260923 17:31:21.957669 21816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:31:21.958899 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:21.958425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60104)
0923 17:31:21.958632 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:21.958755 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
W20260923 17:31:21.959188 21768 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
I20260923 17:31:21.960731 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:21.960399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60120)
0923 17:31:21.960541 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:21.960608 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 17:31:21.960913 21816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37273: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab.
Entry for principal kudu/127.8.29.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab.
I20260923 17:31:21.981503 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.253:37273
--webserver_interface=127.8.29.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab
--principal=kudu/127.8.29.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:46545,127.8.29.253:37273,127.8.29.252:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:31:22.010246 21816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } attempt: 1
I20260923 17:31:22.011345 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.010917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60134)
0923 17:31:22.011103 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.011202 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:31:22.012357 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.012028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60150)
0923 17:31:22.012149 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.012227 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 17:31:22.012560 21816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37273: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
I20260923 17:31:22.080792 21816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } attempt: 2
I20260923 17:31:22.082000 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.081534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60160)
0923 17:31:22.081762 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.081854 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 17:31:22.083002 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.082660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60168)
0923 17:31:22.082779 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.082864 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260923 17:31:22.083206 21816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37273: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
I20260923 17:31:22.174428 21816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } attempt: 3
I20260923 17:31:22.175590 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.175149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60172)
0923 17:31:22.175338 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.175456 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 17:31:22.176556 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.176217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60188)
0923 17:31:22.176344 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.176424 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:31:22.176730 21816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37273: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
W20260923 17:31:22.301209 21823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:22.301550 21823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:22.301671 21823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:22.309247 21823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:22.309370 21823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:22.309422 21823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:22.309499 21823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:22.309553 21823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:22.319902 21823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:46545,127.8.29.253:37273,127.8.29.252:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab
--principal=kudu/127.8.29.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.253:37273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.29.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:22.319959 21816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } attempt: 4
I20260923 17:31:22.321192 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.320642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60204)
0923 17:31:22.320970 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.321060 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260923 17:31:22.321556 21823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:22.322219 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.321859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60218)
0923 17:31:22.322006 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.322097 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:31:22.322410 21816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37273: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37273: connect: Connection refused (error 111)
I20260923 17:31:22.323108 21823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:22.334769 21828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:22.334796 21829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:22.334980 21831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:22.335668 21823 server_base.cc:1034] running on GCE node
I20260923 17:31:22.336481 21823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:22.337983 21823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:22.339176 21823 hybrid_clock.cc:648] HybridClock initialized: now 1790184682339122 us; error 48 us; skew 500 ppm
I20260923 17:31:22.339514 21823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184682, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:22.342561 21823 init.cc:377] Logged in from keytab as kudu/127.8.29.253@KRBTEST.COM (short username kudu)
I20260923 17:31:22.344048 21823 webserver.cc:460] Webserver started at http://127.8.29.253:40305/ using document root <none> and password file <none>
I20260923 17:31:22.344707 21823 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:22.344801 21823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:22.345063 21823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:22.348019 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/data/instance:
uuid: "865552b09671423fad8e208b530ddad5"
format_stamp: "Formatted at 2026-09-23 17:31:22 on dist-test-slave-p7jt"
server_key: "5632eb5f50539ea7f80b41113962a14c"
server_key_iv: "f2f417dbbe96128ec4176002ba2c8195"
server_key_version: "encryptionkey@0"
I20260923 17:31:22.348760 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/wal/instance:
uuid: "865552b09671423fad8e208b530ddad5"
format_stamp: "Formatted at 2026-09-23 17:31:22 on dist-test-slave-p7jt"
server_key: "5632eb5f50539ea7f80b41113962a14c"
server_key_iv: "f2f417dbbe96128ec4176002ba2c8195"
server_key_version: "encryptionkey@0"
I20260923 17:31:22.353932 21823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:31:22.357458 21838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:22.359156 21823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:31:22.359328 21823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/wal
uuid: "865552b09671423fad8e208b530ddad5"
format_stamp: "Formatted at 2026-09-23 17:31:22 on dist-test-slave-p7jt"
server_key: "5632eb5f50539ea7f80b41113962a14c"
server_key_iv: "f2f417dbbe96128ec4176002ba2c8195"
server_key_version: "encryptionkey@0"
I20260923 17:31:22.359519 21823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:22.380160 21823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:22.384616 21823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:22.384862 21823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:22.402546 21823 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.253:37273
I20260923 17:31:22.402588 21889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.253:37273 every 8 connection(s)
I20260923 17:31:22.403993 21823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
I20260923 17:31:22.407927 21890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:22.412205 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 21823
I20260923 17:31:22.412400 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-1/wal/instance
I20260923 17:31:22.412961 8311 external_mini_cluster.cc:1428] Setting key 7c18c1757a79b48dd2216b3b13488b66
I20260923 17:31:22.416092 21890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.29.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.252@KRBTEST.COM" created.
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184682, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:22.459297 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.419607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55948 (local address 127.8.29.254:46545)
0923 17:31:22.419828 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:22.419835 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:22.421060 (+ 1225us) server_negotiation.cc:407] Connection header received
0923 17:31:22.423593 (+ 2533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:22.423598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:22.423674 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:22.423772 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:22.425816 (+ 2044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.426603 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.428753 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.429192 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.429308 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.453463 (+ 24155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:22.453469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:22.453473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:22.453513 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:22.456163 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.456993 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.456999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.457002 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.457131 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.457568 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.457571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.457573 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.457917 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:22.458043 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:22.458845 (+ 802us) server_negotiation.cc:299] Negotiation successful
0923 17:31:22.459068 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:31:22.460425 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.420049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46545 (local address 127.0.0.1:55948)
0923 17:31:22.420871 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.420925 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:22.423260 (+ 2335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:22.424064 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:22.424075 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:22.424802 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:22.425587 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.425612 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.426784 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.426790 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.428596 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.428611 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.429420 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.429429 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.429714 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.449853 (+ 20139us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:22.449920 (+ 67us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:22.453101 (+ 3181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:22.456342 (+ 3241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.456351 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.456407 (+ 56us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.456850 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.457250 (+ 400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.457253 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.457256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.457483 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.458030 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:22.458036 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:22.458437 (+ 401us) client_negotiation.cc:769] Sending connection context
0923 17:31:22.458754 (+ 317us) client_negotiation.cc:240] Negotiation successful
0923 17:31:22.459193 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":346,"threads_started":1}
I20260923 17:31:22.462185 21890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab.
Entry for principal kudu/127.8.29.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab.
I20260923 17:31:22.463938 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.252:39245
--webserver_interface=127.8.29.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab
--principal=kudu/127.8.29.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:46545,127.8.29.253:37273,127.8.29.252:39245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184682, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:31:22.476989 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.463024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60232)
0923 17:31:22.463247 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.463267 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:22.463394 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:22.464453 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:22.464457 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:22.464897 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:22.465080 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.465091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.466443 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.466448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.467688 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.467702 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.468521 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.468530 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.468719 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.469278 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:22.469297 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:22.471620 (+ 2323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:22.474561 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.474566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.474571 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.475001 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.475339 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.475344 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.475348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.475486 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.476049 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:22.476054 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:22.476208 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 17:31:22.476320 (+ 112us) client_negotiation.cc:240] Negotiation successful
0923 17:31:22.476595 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 17:31:22.477061 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.463314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60232 (local address 127.8.29.253:37273)
0923 17:31:22.463961 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:22.463969 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:22.463999 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:31:22.464054 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:22.464062 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:22.464296 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:22.464432 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:22.465227 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.466303 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.467813 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.468368 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.468470 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.471814 (+ 3344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:22.471827 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:22.471841 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:22.471877 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:22.474427 (+ 2550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.475120 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.475128 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.475130 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.475241 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.475578 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.475583 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.475586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.475933 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:22.476126 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:22.476497 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 17:31:22.476706 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":358,"threads_started":1}
I20260923 17:31:22.479921 21890 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:31:22.481082 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.480558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46182)
0923 17:31:22.480786 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.480911 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260923 17:31:22.481329 21839 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
I20260923 17:31:22.482900 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.482565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46186)
0923 17:31:22.482700 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.482792 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 17:31:22.483088 21890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:39245: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
I20260923 17:31:22.532418 21890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } attempt: 1
I20260923 17:31:22.533715 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.533212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46200)
0923 17:31:22.533446 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.533558 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260923 17:31:22.534807 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.534410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46208)
0923 17:31:22.534588 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.534679 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260923 17:31:22.535015 21890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:39245: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
I20260923 17:31:22.603247 21890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } attempt: 2
I20260923 17:31:22.604405 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.603937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46222)
0923 17:31:22.604134 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.604225 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 17:31:22.605410 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.605033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46230)
0923 17:31:22.605177 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.605248 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:31:22.605587 21890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:39245: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
I20260923 17:31:22.621708 21816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } attempt: 5
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184681, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:31:22.636641 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.622595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60234)
0923 17:31:22.622846 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.622871 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:22.623015 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:22.623363 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:22.623368 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:22.623902 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:22.624124 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.624137 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.625206 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.625211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.626622 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.626639 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.627445 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.627452 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.627630 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.628254 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:22.628278 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:22.630977 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:22.634013 (+ 3036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.634019 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.634023 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.634569 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.634998 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.635003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.635007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.635158 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.635690 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:22.635694 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:22.635853 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 17:31:22.635979 (+ 126us) client_negotiation.cc:240] Negotiation successful
0923 17:31:22.636274 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260923 17:31:22.636641 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.622804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60234 (local address 127.8.29.253:37273)
0923 17:31:22.622975 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:22.622979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:22.623000 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:31:22.623130 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:22.623135 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:22.623215 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:22.623296 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:22.624303 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.625065 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.626770 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.627267 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.627374 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.631227 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:22.631232 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:22.631236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:22.631275 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:22.633831 (+ 2556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.634703 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.634707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.634709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.634873 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.635269 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.635273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.635275 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.635574 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:22.635705 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:22.636109 (+ 404us) server_negotiation.cc:299] Negotiation successful
0923 17:31:22.636316 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:31:22.637709 21816 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:31:22.638752 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.638345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46240)
0923 17:31:22.638534 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.638612 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 17:31:22.639635 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.639340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46246)
0923 17:31:22.639470 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.639535 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260923 17:31:22.639845 21816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:39245: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
I20260923 17:31:22.691077 21816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } attempt: 1
I20260923 17:31:22.692224 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.691750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46248)
0923 17:31:22.691963 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.692061 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260923 17:31:22.693410 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.693034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46252)
0923 17:31:22.693189 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.693288 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:31:22.693609 21816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:39245: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
I20260923 17:31:22.696830 21890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } attempt: 3
I20260923 17:31:22.697986 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.697419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46258)
0923 17:31:22.697664 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.697781 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:31:22.699147 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.698668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46262)
0923 17:31:22.698912 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.699007 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260923 17:31:22.699399 21890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:39245: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
I20260923 17:31:22.761863 21816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } attempt: 2
I20260923 17:31:22.763098 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.762614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46266)
0923 17:31:22.762827 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.762937 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260923 17:31:22.764171 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.763850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46270)
0923 17:31:22.763980 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.764060 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 17:31:22.764364 21816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:39245: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
W20260923 17:31:22.809808 21896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:22.810098 21896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:22.810210 21896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:22.817473 21896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:22.817579 21896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:22.817626 21896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:22.817706 21896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:22.817760 21896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:22.827715 21896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:46545,127.8.29.253:37273,127.8.29.252:39245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab
--principal=kudu/127.8.29.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.252:39245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.29.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:22.829049 21896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:22.830303 21896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:22.841989 21902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:22.841953 21905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:22.842068 21903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:22.842598 21890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } attempt: 4
I20260923 17:31:22.842739 21896 server_base.cc:1034] running on GCE node
I20260923 17:31:22.843690 21896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:22.843747 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.843269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46282)
0923 17:31:22.843516 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.843602 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260923 17:31:22.844772 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.844387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46288)
0923 17:31:22.844555 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.844639 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:31:22.844996 21890 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:39245: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
I20260923 17:31:22.845429 21896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:22.846621 21896 hybrid_clock.cc:648] HybridClock initialized: now 1790184682846559 us; error 49 us; skew 500 ppm
I20260923 17:31:22.846982 21896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184682, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:22.849951 21896 init.cc:377] Logged in from keytab as kudu/127.8.29.252@KRBTEST.COM (short username kudu)
I20260923 17:31:22.851425 21896 webserver.cc:460] Webserver started at http://127.8.29.252:41493/ using document root <none> and password file <none>
I20260923 17:31:22.852085 21896 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:22.852180 21896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:22.852429 21896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:22.855499 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/data/instance:
uuid: "7856d7f88fef4cf9b44fc04f8c79d235"
format_stamp: "Formatted at 2026-09-23 17:31:22 on dist-test-slave-p7jt"
server_key: "ebab1526c5a2423d29c40b6c830ec1fa"
server_key_iv: "b6c4fddace238766c92c5551666be606"
server_key_version: "encryptionkey@0"
I20260923 17:31:22.856335 21896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/wal/instance:
uuid: "7856d7f88fef4cf9b44fc04f8c79d235"
format_stamp: "Formatted at 2026-09-23 17:31:22 on dist-test-slave-p7jt"
server_key: "ebab1526c5a2423d29c40b6c830ec1fa"
server_key_iv: "b6c4fddace238766c92c5551666be606"
server_key_version: "encryptionkey@0"
I20260923 17:31:22.861428 21896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:31:22.865082 21912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:22.866645 21896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:31:22.866823 21896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/wal
uuid: "7856d7f88fef4cf9b44fc04f8c79d235"
format_stamp: "Formatted at 2026-09-23 17:31:22 on dist-test-slave-p7jt"
server_key: "ebab1526c5a2423d29c40b6c830ec1fa"
server_key_iv: "b6c4fddace238766c92c5551666be606"
server_key_version: "encryptionkey@0"
I20260923 17:31:22.867019 21896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:22.870545 21816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } attempt: 3
I20260923 17:31:22.871691 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.871148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46302)
0923 17:31:22.871399 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.871504 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 17:31:22.872886 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.872396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46310)
0923 17:31:22.872645 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.872745 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20260923 17:31:22.873132 21816 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:39245: Network error: Client connection negotiation failed: client connection to 127.8.29.252:39245: connect: Connection refused (error 111)
I20260923 17:31:22.900728 21896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:22.905541 21896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:22.905869 21896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:22.924906 21896 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.252:39245
I20260923 17:31:22.924934 21963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.252:39245 every 8 connection(s)
I20260923 17:31:22.926391 21896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
I20260923 17:31:22.927913 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 21896
I20260923 17:31:22.928133 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790184478467572-8311-0/minicluster-data/master-2/wal/instance
I20260923 17:31:22.928604 8311 external_mini_cluster.cc:1428] Setting key c1813f0cef88681703ee2146a924ebd0
I20260923 17:31:22.930965 21964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:22.938652 21964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184682, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:22.952841 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.930657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46314 (local address 127.8.29.252:39245)
0923 17:31:22.931555 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:22.931573 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:22.931882 (+ 309us) server_negotiation.cc:407] Connection header received
0923 17:31:22.933699 (+ 1817us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:22.933723 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:22.934166 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:22.934620 (+ 454us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:22.936198 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.937537 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.939042 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.939848 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.940031 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.944946 (+ 4915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:22.944956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:22.944972 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:22.945022 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:22.948439 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.949315 (+ 876us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.949321 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.949328 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.949499 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.949996 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.950002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.950005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.950647 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:22.950985 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:22.951278 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 17:31:22.951715 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":390,"threads_started":1}
I20260923 17:31:22.956441 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.941322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:46545 (local address 127.0.0.1:55962)
0923 17:31:22.942113 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.942151 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:22.942337 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:22.942744 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:22.942752 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:22.943472 (+ 720us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:22.943881 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.943897 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.944903 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.944907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.946407 (+ 1500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.946423 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.947423 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.947428 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.947557 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.948099 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:22.948136 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:22.950530 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:22.953822 (+ 3292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.953834 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.953843 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.954308 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.954725 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.954730 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.954734 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.954886 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.955434 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:22.955442 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:22.955614 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 17:31:22.955821 (+ 207us) client_negotiation.cc:240] Negotiation successful
0923 17:31:22.956134 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":413,"threads_started":1}
I20260923 17:31:22.957345 21819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.941273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55962 (local address 127.8.29.254:46545)
0923 17:31:22.941456 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:22.941461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:22.942291 (+ 830us) server_negotiation.cc:407] Connection header received
0923 17:31:22.942491 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:22.942498 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:22.942571 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:22.942657 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:22.944035 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.944755 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.946576 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.947226 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.947359 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.950742 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:22.950748 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:22.950753 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:22.950796 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:22.953668 (+ 2872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.954431 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.954437 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.954441 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.954617 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.954990 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.954995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.954999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.955313 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:22.955468 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:22.956773 (+ 1305us) server_negotiation.cc:299] Negotiation successful
0923 17:31:22.956999 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:31:22.958668 21964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184682, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:31:22.971753 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.955824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55974 (local address 127.8.29.254:46545)
0923 17:31:22.956326 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:22.956330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:22.956348 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:31:22.956385 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:22.956388 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:22.956438 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:22.956519 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:22.958328 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.959314 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.960776 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.961197 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.961308 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.966077 (+ 4769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:22.966082 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:22.966085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:22.966125 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:22.969088 (+ 2963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.969789 (+ 701us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.969793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.969796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.969941 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.970336 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.970341 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.970344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.970710 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:22.970963 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:22.971258 (+ 295us) server_negotiation.cc:299] Negotiation successful
0923 17:31:22.971442 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":239,"threads_started":1}
I20260923 17:31:22.972723 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.959313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37273 (local address 127.0.0.1:60240)
0923 17:31:22.959488 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.959514 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:22.959613 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:22.960084 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:22.960088 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:22.960450 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:22.960596 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.960605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.961607 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.961611 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.962801 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.962815 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.963855 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.963860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.964036 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.964635 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:22.964659 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:22.966819 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:22.969922 (+ 3103us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.969927 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.969931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.970364 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.970913 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.970918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.970922 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.971096 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.971773 (+ 677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:22.971778 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:22.971948 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 17:31:22.972088 (+ 140us) client_negotiation.cc:240] Negotiation successful
0923 17:31:22.972374 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 17:31:22.972795 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.959545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60240 (local address 127.8.29.253:37273)
0923 17:31:22.959746 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:22.959752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:22.959770 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:31:22.959837 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:22.959843 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:22.959917 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:22.960018 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:22.960751 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.961464 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.962954 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.963656 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.963795 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.967011 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:22.967019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:22.967023 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:22.967066 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:22.969792 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.970561 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.970581 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.970585 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.970740 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.971214 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.971219 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.971223 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.971569 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:22.971712 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:22.972201 (+ 489us) server_negotiation.cc:299] Negotiation successful
0923 17:31:22.972423 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:31:22.973982 21964 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184682, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
Sep 23 17:31:22 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:31:22.987131 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.974712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46328)
0923 17:31:22.974882 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:22.974898 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:22.974985 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:22.975310 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:22.975313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:22.975687 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:22.975832 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.975840 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.976771 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.976776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.977614 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:22.977625 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.978390 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.978397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:22.978593 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.979091 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:22.979106 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:22.981181 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:22.984646 (+ 3465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.984652 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.984656 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.985024 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.985436 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:22.985439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:22.985442 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:22.985562 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:22.986119 (+ 557us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:22.986124 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:22.986268 (+ 144us) client_negotiation.cc:769] Sending connection context
0923 17:31:22.986396 (+ 128us) client_negotiation.cc:240] Negotiation successful
0923 17:31:22.986718 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260923 17:31:22.987208 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.974891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.8.29.252:39245)
0923 17:31:22.975050 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:22.975056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:22.975073 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:31:22.975117 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:22.975122 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:22.975187 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:22.975285 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:22.975969 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.976650 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.977745 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.978227 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.978315 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.981412 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:22.981419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:22.981424 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:22.981461 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:22.984494 (+ 3033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.985150 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.985157 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.985160 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.985314 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.985658 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.985664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.985667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.986030 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:22.986157 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:22.986513 (+ 356us) server_negotiation.cc:299] Negotiation successful
0923 17:31:22.986890 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":144,"server-negotiator.queue_time_us":52}
I20260923 17:31:22.988897 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:22.974052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60248 (local address 127.8.29.253:37273)
0923 17:31:22.974202 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:22.974207 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:22.974225 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:31:22.974327 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:22.974332 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:22.974402 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:22.974531 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:22.975994 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.976676 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.977881 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:22.978325 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:22.978442 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:22.982912 (+ 4470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:22.982919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:22.982924 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:22.982962 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:22.985936 (+ 2974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.986735 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.986739 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.986742 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.986914 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:22.987345 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:22.987351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:22.987355 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:22.987708 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:22.987951 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:22.988311 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 17:31:22.988579 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:31:22.992442 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235: Bootstrap starting.
I20260923 17:31:22.995901 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:31:22.996115 21964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:22.997401 21964 log.cc:824] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:23.001293 21964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235: No bootstrap required, opened a new log
I20260923 17:31:23.012576 21964 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } }
I20260923 17:31:23.014082 21964 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:23.014317 21964 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7856d7f88fef4cf9b44fc04f8c79d235, State: Initialized, Role: FOLLOWER
I20260923 17:31:23.015288 21964 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } }
I20260923 17:31:23.017285 21978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.017735 21978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:31:23.018831 21964 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:31:23.022243 21895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:23.002279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60258 (local address 127.8.29.253:37273)
0923 17:31:23.002673 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:23.002680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:23.003208 (+ 528us) server_negotiation.cc:407] Connection header received
0923 17:31:23.003556 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:23.003562 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:23.003646 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:23.003719 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:23.006001 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.007224 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.009550 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.010293 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.010409 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:23.014412 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:23.014420 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:23.014424 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:23.014496 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:23.018269 (+ 3773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.019244 (+ 975us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.019249 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.019252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.019417 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.019868 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.019873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.019876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.020226 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:23.021348 (+ 1122us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:23.021498 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 17:31:23.021903 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252}
I20260923 17:31:23.022377 21969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:23.001782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55980 (local address 127.8.29.254:46545)
0923 17:31:23.002004 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:23.002011 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:23.002678 (+ 667us) server_negotiation.cc:407] Connection header received
0923 17:31:23.003069 (+ 391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:23.003075 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:23.003162 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:23.003240 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:23.005440 (+ 2200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.006532 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.008733 (+ 2201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.009389 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.009490 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:23.013792 (+ 4302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:23.013800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:23.013804 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:23.013856 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:23.019084 (+ 5228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.019873 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.019878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.019881 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.020024 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.020491 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.020496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.020500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.020847 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:23.021053 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:23.021842 (+ 789us) server_negotiation.cc:299] Negotiation successful
0923 17:31:23.022067 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260923 17:31:23.029528 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:23.003760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.8.29.252:39245)
0923 17:31:23.003980 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:23.003987 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:23.013815 (+ 9828us) server_negotiation.cc:407] Connection header received
0923 17:31:23.013999 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:23.014004 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:23.014088 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:23.014180 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:23.015851 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.017030 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.018879 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.019546 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.019696 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:23.023274 (+ 3578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:23.023281 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:23.023285 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:23.023325 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:23.026579 (+ 3254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.027352 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.027357 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.027361 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.027536 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.027966 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.027971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.027974 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.028300 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:23.028582 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:23.028898 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 17:31:23.029119 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":74}
W20260923 17:31:23.032754 21989 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:31:23.032895 21989 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:31:23.029503 21896 master_runner.cc:186] Error getting master registration for 127.8.29.254:46545: OK, instance_id { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" instance_seqno: 1790184681913964 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:31:23.038733 21896 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:31:23.050377 21816 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } attempt: 4
Sep 23 17:31:23 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184681, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
I20260923 17:31:23.064963 21818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:23.051047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46348)
0923 17:31:23.051253 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:23.051272 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:23.051404 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:23.051751 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:23.051755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:23.052260 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:23.052483 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:23.052497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.053979 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.053984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:23.054971 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:23.054985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.055712 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.055718 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:23.055884 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:23.056459 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:23.056477 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:23.058559 (+ 2082us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:23.062353 (+ 3794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:23.062358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:23.062363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:23.062901 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:23.063344 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:23.063349 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:23.063353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:23.063508 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:23.064142 (+ 634us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:23.064145 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:23.064296 (+ 151us) client_negotiation.cc:769] Sending connection context
0923 17:31:23.064414 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:31:23.064701 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 17:31:23.065369 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:23.051083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46348 (local address 127.8.29.252:39245)
0923 17:31:23.051258 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:23.051265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:23.051347 (+ 82us) server_negotiation.cc:407] Connection header received
0923 17:31:23.051521 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:23.051526 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:23.051621 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:23.051702 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:23.052655 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.053806 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.055114 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.055569 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.055683 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:23.058744 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:23.058749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:23.058752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:23.058784 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:23.062185 (+ 3401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.063012 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.063018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.063022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.063204 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.063623 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.063628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.063631 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.064037 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:23.064185 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:23.064551 (+ 366us) server_negotiation.cc:299] Negotiation successful
0923 17:31:23.064804 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:31:23.069792 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c: Bootstrap starting.
I20260923 17:31:23.073042 21816 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:23.074496 21816 log.cc:824] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:23.077550 21816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c: No bootstrap required, opened a new log
I20260923 17:31:23.085877 21816 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } }
I20260923 17:31:23.086191 21816 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:23.086302 21816 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28fa207e24cf4ef1b11225f7aad6f82c, State: Initialized, Role: FOLLOWER
I20260923 17:31:23.086891 21816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } }
I20260923 17:31:23.087743 21992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.088131 21992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:31:23.088986 21816 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:31:23.101706 22003 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:31:23.101858 22003 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:31:23.097885 21749 master_runner.cc:186] Error getting master registration for 127.8.29.253:37273: OK, instance_id { permanent_uuid: "865552b09671423fad8e208b530ddad5" instance_seqno: 1790184682397225 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:31:23.106590 21749 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:31:23.144218 21890 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } attempt: 5
W20260923 17:31:23.153996 21770 tablet.cc:2369] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 23 17:31:23 dist-test-slave-p7jt krb5kdc[21736](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184682, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
W20260923 17:31:23.158206 21918 tablet.cc:2369] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:31:23.163551 21892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:23.144794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:39245 (local address 127.0.0.1:46364)
0923 17:31:23.145109 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:23.145126 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:23.145243 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:23.145564 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:23.145568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:23.145989 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:23.146144 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:23.146153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.147170 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.147176 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:23.148041 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:23.148053 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.148768 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.148774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:23.148989 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:23.153114 (+ 4125us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:23.153133 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:23.155636 (+ 2503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:23.161032 (+ 5396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:23.161036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:23.161039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:23.161454 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:23.161818 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:23.161822 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:23.161824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:23.161962 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:23.162501 (+ 539us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:23.162506 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:23.162686 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 17:31:23.162821 (+ 135us) client_negotiation.cc:240] Negotiation successful
0923 17:31:23.163156 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210}
I20260923 17:31:23.163997 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:23.144876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46364 (local address 127.8.29.252:39245)
0923 17:31:23.145068 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:23.145075 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:23.145197 (+ 122us) server_negotiation.cc:407] Connection header received
0923 17:31:23.145351 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:23.145361 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:23.145431 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:23.145513 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:23.146290 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.147045 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.148178 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:23.148632 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:23.148738 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:23.155826 (+ 7088us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:23.155832 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:23.155835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:23.155870 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:23.160885 (+ 5015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.161572 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.161578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.161582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.161706 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:23.162069 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:23.162074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:23.162076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:23.162362 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:23.162540 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:23.162953 (+ 413us) server_negotiation.cc:299] Negotiation successful
0923 17:31:23.163281 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:31:23.168449 21890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5: Bootstrap starting.
I20260923 17:31:23.171953 21890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:23.173081 21890 log.cc:824] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:23.176028 21890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5: No bootstrap required, opened a new log
I20260923 17:31:23.184094 21890 raft_consensus.cc:348] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } }
I20260923 17:31:23.184417 21890 raft_consensus.cc:374] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:23.184522 21890 raft_consensus.cc:729] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 865552b09671423fad8e208b530ddad5, State: Initialized, Role: FOLLOWER
I20260923 17:31:23.185369 21890 consensus_queue.cc:260] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } }
I20260923 17:31:23.186350 22006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.186872 22006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:31:23.187646 21890 sys_catalog.cc:564] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:31:23.203303 21823 master_runner.cc:263] No leader master found from master 865552b09671423fad8e208b530ddad5
I20260923 17:31:23.203473 21823 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 17:31:23.206346 22017 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:31:23.206789 22017 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:31:23.324713 21978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:31:23.325070 21978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } }
I20260923 17:31:23.326930 21978 leader_election.cc:288] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 28fa207e24cf4ef1b11225f7aad6f82c (127.8.29.254:46545), 865552b09671423fad8e208b530ddad5 (127.8.29.253:37273)
I20260923 17:31:23.328181 21791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" is_pre_election: true
I20260923 17:31:23.328812 21791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7856d7f88fef4cf9b44fc04f8c79d235 in term 0.
I20260923 17:31:23.330191 21913 leader_election.cc:302] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fa207e24cf4ef1b11225f7aad6f82c, 7856d7f88fef4cf9b44fc04f8c79d235; no voters:
I20260923 17:31:23.331712 21865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865552b09671423fad8e208b530ddad5" is_pre_election: true
I20260923 17:31:23.332293 21865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7856d7f88fef4cf9b44fc04f8c79d235 in term 0.
I20260923 17:31:23.332789 21978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:31:23.332947 21978 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:31:23.333046 21978 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:23.338343 21978 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } }
I20260923 17:31:23.339743 21791 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "28fa207e24cf4ef1b11225f7aad6f82c"
I20260923 17:31:23.339994 21791 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:23.343604 21791 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7856d7f88fef4cf9b44fc04f8c79d235 in term 1.
I20260923 17:31:23.344214 21913 leader_election.cc:302] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 28fa207e24cf4ef1b11225f7aad6f82c, 7856d7f88fef4cf9b44fc04f8c79d235; no voters:
I20260923 17:31:23.352401 21865 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "865552b09671423fad8e208b530ddad5"
I20260923 17:31:23.352670 21865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:23.356197 21865 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7856d7f88fef4cf9b44fc04f8c79d235 in term 1.
I20260923 17:31:23.358215 21978 leader_election.cc:288] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [CANDIDATE]: Term 1 election: Requested vote from peers 28fa207e24cf4ef1b11225f7aad6f82c (127.8.29.254:46545), 865552b09671423fad8e208b530ddad5 (127.8.29.253:37273)
I20260923 17:31:23.358713 21978 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:31:23.359177 21978 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [term 1 LEADER]: Becoming Leader. State: Replica: 7856d7f88fef4cf9b44fc04f8c79d235, State: Running, Role: LEADER
I20260923 17:31:23.360085 21978 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } }
I20260923 17:31:23.371196 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7856d7f88fef4cf9b44fc04f8c79d235. Latest consensus state: current_term: 1 leader_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.371726 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:23.383648 22021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:31:23.385882 22021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260923 17:31:23.387513 21844 tablet.cc:2369] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:31:23.418145 21865 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [term 1 FOLLOWER]: Refusing update from remote peer 7856d7f88fef4cf9b44fc04f8c79d235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:23.419368 22019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:23.425755 21791 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [term 1 FOLLOWER]: Refusing update from remote peer 7856d7f88fef4cf9b44fc04f8c79d235: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:23.426774 22019 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [LEADER]: Connected to new peer: Peer: permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:23.470445 21992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7856d7f88fef4cf9b44fc04f8c79d235. Latest consensus state: current_term: 1 leader_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.470854 21992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:31:23.476373 22022 mvcc.cc:205] Tried to move back new op lower bound from 7332596463226650624 to 7332596463048523776. Current Snapshot: MvccSnapshot[applied={T|T < 7332596463226650624}]
I20260923 17:31:23.477164 22006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7856d7f88fef4cf9b44fc04f8c79d235. Latest consensus state: current_term: 1 leader_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.477510 22006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:31:23.479290 22024 mvcc.cc:205] Tried to move back new op lower bound from 7332596463226650624 to 7332596463048523776. Current Snapshot: MvccSnapshot[applied={T|T < 7332596463226650624}]
I20260923 17:31:23.483841 22023 mvcc.cc:205] Tried to move back new op lower bound from 7332596463226650624 to 7332596463048523776. Current Snapshot: MvccSnapshot[applied={T|T < 7332596463226650624}]
I20260923 17:31:23.490698 22006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.490978 22006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:31:23.492731 21992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.493000 21992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:31:23.495065 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.495338 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:23.495703 22019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "28fa207e24cf4ef1b11225f7aad6f82c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46545 } } peers { permanent_uuid: "865552b09671423fad8e208b530ddad5" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37273 } } peers { permanent_uuid: "7856d7f88fef4cf9b44fc04f8c79d235" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 39245 } } }
I20260923 17:31:23.495918 22019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:23.498761 22021 catalog_manager.cc:1347] Generated new cluster ID: c0a37814710240318bc9dc5d45dc8934
I20260923 17:31:23.498888 22021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:31:23.565982 22021 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:31:23.567914 22021 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:31:23.586205 22021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7856d7f88fef4cf9b44fc04f8c79d235: Generated new TSK 0
I20260923 17:31:23.587423 22021 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:31:24.103902 22003 catalog_manager.cc:1259] Loaded cluster ID: c0a37814710240318bc9dc5d45dc8934
I20260923 17:31:24.104003 22003 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c: loading cluster ID for follower catalog manager: success
I20260923 17:31:24.106380 22003 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c: acquiring CA information for follower catalog manager: success
I20260923 17:31:24.108150 22003 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 28fa207e24cf4ef1b11225f7aad6f82c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:31:24.208963 22017 catalog_manager.cc:1259] Loaded cluster ID: c0a37814710240318bc9dc5d45dc8934
I20260923 17:31:24.209069 22017 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5: loading cluster ID for follower catalog manager: success
I20260923 17:31:24.211488 22017 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5: acquiring CA information for follower catalog manager: success
I20260923 17:31:24.213227 22017 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 865552b09671423fad8e208b530ddad5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:31:24.672259 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 21749
I20260923 17:31:24.673591 21811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:24.782759 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 21749
I20260923 17:31:24.800650 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 21823
I20260923 17:31:24.801802 21885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:24.907325 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 21823
I20260923 17:31:24.924729 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 21896
I20260923 17:31:24.925944 21959 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:25.034556 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 21896
2026-09-23T17:31:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4807 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:31:25 dist-test-slave-p7jt krb5kdc[22042](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:31:25 dist-test-slave-p7jt krb5kdc[22042](info): set up 2 sockets
Sep 23 17:31:25 dist-test-slave-p7jt krb5kdc[22042](info): commencing operation
krb5kdc: starting...
W20260923 17:31:26.046847 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.975s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:31:26 dist-test-slave-p7jt krb5kdc[22042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:31:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:31:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:31:26.275063 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44057
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:44057
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:26.598037 22055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:26.598320 22055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:26.598475 22055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:26.605743 22055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:26.605863 22055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:26.605940 22055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:26.606024 22055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:26.606082 22055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:26.616147 22055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:44057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:26.617440 22055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:26.618847 22055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:26.630635 22060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:26.630695 22061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:26.630673 22063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:26.631548 22055 server_base.cc:1034] running on GCE node
I20260923 17:31:26.632355 22055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:26.633849 22055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:26.635056 22055 hybrid_clock.cc:648] HybridClock initialized: now 1790184686635003 us; error 48 us; skew 500 ppm
I20260923 17:31:26.635435 22055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:26 dist-test-slave-p7jt krb5kdc[22042](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184686, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:26.638418 22055 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:31:26.639923 22055 webserver.cc:460] Webserver started at http://127.8.29.254:44201/ using document root <none> and password file <none>
I20260923 17:31:26.640574 22055 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:26.640668 22055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:26.640933 22055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:26.643941 22055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "3c7a01ae2dac48d09123fbdf3486bf14"
format_stamp: "Formatted at 2026-09-23 17:31:26 on dist-test-slave-p7jt"
server_key: "4b39d1a8e90db62da93d4a84e155a622"
server_key_iv: "32b074551165a693e0b4d06e8e86a0aa"
server_key_version: "encryptionkey@0"
I20260923 17:31:26.644682 22055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "3c7a01ae2dac48d09123fbdf3486bf14"
format_stamp: "Formatted at 2026-09-23 17:31:26 on dist-test-slave-p7jt"
server_key: "4b39d1a8e90db62da93d4a84e155a622"
server_key_iv: "32b074551165a693e0b4d06e8e86a0aa"
server_key_version: "encryptionkey@0"
I20260923 17:31:26.650352 22055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:31:26.654119 22070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:26.655673 22055 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:31:26.655851 22055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "3c7a01ae2dac48d09123fbdf3486bf14"
format_stamp: "Formatted at 2026-09-23 17:31:26 on dist-test-slave-p7jt"
server_key: "4b39d1a8e90db62da93d4a84e155a622"
server_key_iv: "32b074551165a693e0b4d06e8e86a0aa"
server_key_version: "encryptionkey@0"
I20260923 17:31:26.656046 22055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:26.676784 22055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:26.681187 22055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:26.681458 22055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:26.699029 22055 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:44057
I20260923 17:31:26.699060 22121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:44057 every 8 connection(s)
I20260923 17:31:26.700433 22055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:31:26.704326 22122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:26.706859 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 22055
I20260923 17:31:26.707074 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:31:26.707748 8311 external_mini_cluster.cc:1428] Setting key 6113fb82c3279c07831760aecb7f8c08
I20260923 17:31:26.716795 22122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14: Bootstrap starting.
I20260923 17:31:26.720781 22122 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:26.722168 22122 log.cc:824] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:31:26 dist-test-slave-p7jt krb5kdc[22042](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:26.725404 22122 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14: No bootstrap required, opened a new log
I20260923 17:31:26.731612 22125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:26.709713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50686 (local address 127.8.29.254:44057)
0923 17:31:26.710561 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:26.710574 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:26.710616 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:31:26.712317 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:26.712344 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:26.712786 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:26.713164 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:26.715039 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:26.716357 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:26.718253 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:26.719041 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:26.719177 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:26.724284 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:26.724301 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:26.724318 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:26.724357 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:26.727419 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:26.728172 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:26.728180 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:26.728189 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:26.728386 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:26.728763 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:26.728766 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:26.728768 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:26.729321 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:26.729629 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:26.730081 (+ 452us) server_negotiation.cc:299] Negotiation successful
0923 17:31:26.730560 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":359,"threads_started":1}
I20260923 17:31:26.733244 22122 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a01ae2dac48d09123fbdf3486bf14" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44057 } }
I20260923 17:31:26.733599 22122 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:26.733698 22122 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7a01ae2dac48d09123fbdf3486bf14, State: Initialized, Role: FOLLOWER
I20260923 17:31:26.734314 22122 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a01ae2dac48d09123fbdf3486bf14" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44057 } }
I20260923 17:31:26.734661 22122 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:31:26.734810 22122 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:31:26.735000 22122 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:26.735558 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:31:26.738049 22122 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a01ae2dac48d09123fbdf3486bf14" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44057 } }
I20260923 17:31:26.738685 22122 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c7a01ae2dac48d09123fbdf3486bf14; no voters:
I20260923 17:31:26.739300 22122 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:31:26.739432 22127 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:31:26.740106 22127 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [term 1 LEADER]: Becoming Leader. State: Replica: 3c7a01ae2dac48d09123fbdf3486bf14, State: Running, Role: LEADER
I20260923 17:31:26.740691 22127 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a01ae2dac48d09123fbdf3486bf14" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44057 } }
I20260923 17:31:26.741580 22122 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:31:26.743419 22128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c7a01ae2dac48d09123fbdf3486bf14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a01ae2dac48d09123fbdf3486bf14" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44057 } } }
I20260923 17:31:26.743815 22128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:26.745383 22133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:31:26.745332 22129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c7a01ae2dac48d09123fbdf3486bf14. Latest consensus state: current_term: 1 leader_uuid: "3c7a01ae2dac48d09123fbdf3486bf14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7a01ae2dac48d09123fbdf3486bf14" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44057 } } }
I20260923 17:31:26.745638 22129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:26.749531 22133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:31:26.758260 22133 catalog_manager.cc:1347] Generated new cluster ID: 88ea7c2c73434a468e7e2de19f50d786
I20260923 17:31:26.758401 22133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:31:26.773033 22133 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:31:26.774355 22133 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:31:26.780433 22133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c7a01ae2dac48d09123fbdf3486bf14: Generated new TSK 0
I20260923 17:31:26.781309 22133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:31:26.796581 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 22055
I20260923 17:31:26.797849 22117 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:26.888362 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 22055
2026-09-23T17:31:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1849 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:31:26 dist-test-slave-p7jt krb5kdc[22150](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:31:26 dist-test-slave-p7jt krb5kdc[22150](info): set up 2 sockets
Sep 23 17:31:26 dist-test-slave-p7jt krb5kdc[22150](info): commencing operation
krb5kdc: starting...
W20260923 17:31:27.902990 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.981s user 0.002s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:31:28 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:31:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:31:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:31:28.135066 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:32825
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:28.466230 22163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:28.466583 22163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:28.466719 22163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:28.473996 22163 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:28.474114 22163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:28.474195 22163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:28.474274 22163 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:28.474334 22163 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:28.484357 22163 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:32825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:28.485719 22163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:28.487200 22163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:28.499014 22168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:28.498941 22169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:28.498991 22171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:28.499648 22163 server_base.cc:1034] running on GCE node
I20260923 17:31:28.500541 22163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:28.502017 22163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:28.503230 22163 hybrid_clock.cc:648] HybridClock initialized: now 1790184688503174 us; error 38 us; skew 500 ppm
I20260923 17:31:28.503579 22163 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:28 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184688, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:28.509855 22163 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:31:28.511356 22163 webserver.cc:460] Webserver started at http://127.8.29.254:46249/ using document root <none> and password file <none>
I20260923 17:31:28.512060 22163 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:28.512180 22163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:28.512480 22163 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:28.515581 22163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "2efa957b67ab41ecb8822193aa2053df"
format_stamp: "Formatted at 2026-09-23 17:31:28 on dist-test-slave-p7jt"
server_key: "b14cf4a70ec6711b503d8391ee4130a6"
server_key_iv: "226d5857b1fcba3fed365cebb92fbdee"
server_key_version: "encryptionkey@0"
I20260923 17:31:28.516333 22163 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "2efa957b67ab41ecb8822193aa2053df"
format_stamp: "Formatted at 2026-09-23 17:31:28 on dist-test-slave-p7jt"
server_key: "b14cf4a70ec6711b503d8391ee4130a6"
server_key_iv: "226d5857b1fcba3fed365cebb92fbdee"
server_key_version: "encryptionkey@0"
I20260923 17:31:28.529294 22163 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.005s sys 0.007s
I20260923 17:31:28.534513 22178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:28.536410 22163 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:31:28.536708 22163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "2efa957b67ab41ecb8822193aa2053df"
format_stamp: "Formatted at 2026-09-23 17:31:28 on dist-test-slave-p7jt"
server_key: "b14cf4a70ec6711b503d8391ee4130a6"
server_key_iv: "226d5857b1fcba3fed365cebb92fbdee"
server_key_version: "encryptionkey@0"
I20260923 17:31:28.536968 22163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:28.553222 22163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:28.557391 22163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:28.557621 22163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:28.574981 22163 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:32825
I20260923 17:31:28.575016 22229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:32825 every 8 connection(s)
I20260923 17:31:28.576344 22163 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:31:28.578372 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 22163
I20260923 17:31:28.578613 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:31:28.579211 8311 external_mini_cluster.cc:1428] Setting key 9b66de8d24ec5b317a17a9bbc46b1a8c
I20260923 17:31:28.580278 22230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:31:28 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:28.593897 22230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Bootstrap starting.
I20260923 17:31:28.597855 22230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:28.599085 22230 log.cc:824] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:28.602296 22230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: No bootstrap required, opened a new log
I20260923 17:31:28.602519 22233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:28.581204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46148 (local address 127.8.29.254:32825)
0923 17:31:28.581964 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:28.581979 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:28.582027 (+ 48us) server_negotiation.cc:407] Connection header received
0923 17:31:28.583748 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:28.583777 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:28.584317 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:28.584762 (+ 445us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:28.586010 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:28.588035 (+ 2025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:28.589308 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:28.590320 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:28.590537 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:28.594481 (+ 3944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:28.594492 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:28.594512 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:28.594555 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:28.598152 (+ 3597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:28.598889 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:28.598897 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:28.598905 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:28.599124 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:28.599483 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:28.599487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:28.599490 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:28.600099 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:28.600330 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:28.600796 (+ 466us) server_negotiation.cc:299] Negotiation successful
0923 17:31:28.601220 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":304,"threads_started":1}
I20260923 17:31:28.610208 22230 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:28.610615 22230 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:28.610702 22230 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efa957b67ab41ecb8822193aa2053df, State: Initialized, Role: FOLLOWER
I20260923 17:31:28.611222 22230 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:28.611428 22230 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:31:28.611523 22230 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:31:28.611646 22230 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:28.613942 22230 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:28.614411 22230 leader_election.cc:302] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2efa957b67ab41ecb8822193aa2053df; no voters:
I20260923 17:31:28.615016 22230 leader_election.cc:288] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:31:28.615137 22235 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:31:28.615731 22235 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 1 LEADER]: Becoming Leader. State: Replica: 2efa957b67ab41ecb8822193aa2053df, State: Running, Role: LEADER
I20260923 17:31:28.616256 22235 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:28.617040 22230 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:31:28.618752 22236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2efa957b67ab41ecb8822193aa2053df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } } }
I20260923 17:31:28.619153 22236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:28.619645 22237 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2efa957b67ab41ecb8822193aa2053df. Latest consensus state: current_term: 1 leader_uuid: "2efa957b67ab41ecb8822193aa2053df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } } }
I20260923 17:31:28.620007 22237 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:28.621217 22240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:31:28.625707 22240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:31:28.634667 22240 catalog_manager.cc:1347] Generated new cluster ID: 8599245691b64b1c8c5541508ae482a5
I20260923 17:31:28.634779 22240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:31:28.646021 22240 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:31:28.648001 22240 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:31:28.668442 22240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Generated new TSK 0
I20260923 17:31:28.669628 22240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:31:28.738699 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:32825
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:29.066180 22256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:29.066471 22256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:29.066596 22256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:29.074002 22256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:29.074221 22256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:29.074404 22256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:31:29.085031 22256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:29.086416 22256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:29.087867 22256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:29.102413 22262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:29.102572 22264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:29.102413 22261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:29.103281 22256 server_base.cc:1034] running on GCE node
I20260923 17:31:29.103909 22256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:29.104868 22256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:29.106086 22256 hybrid_clock.cc:648] HybridClock initialized: now 1790184689106034 us; error 37 us; skew 500 ppm
I20260923 17:31:29.106415 22256 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:29 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184689, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:29.109500 22256 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:31:29.111035 22256 webserver.cc:460] Webserver started at http://127.8.29.193:43429/ using document root <none> and password file <none>
I20260923 17:31:29.111714 22256 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:29.111828 22256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:29.112118 22256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:29.115083 22256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "6679092b2487475e85075d068cbf7ab2"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "9c8dc2e2e5a562a14e9c677a16fa7b65"
server_key_iv: "855b92e3e70909d2a51d555fda3ab12d"
server_key_version: "encryptionkey@0"
I20260923 17:31:29.115825 22256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "6679092b2487475e85075d068cbf7ab2"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "9c8dc2e2e5a562a14e9c677a16fa7b65"
server_key_iv: "855b92e3e70909d2a51d555fda3ab12d"
server_key_version: "encryptionkey@0"
I20260923 17:31:29.121389 22256 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:31:29.124914 22271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:29.126304 22256 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:31:29.126498 22256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "6679092b2487475e85075d068cbf7ab2"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "9c8dc2e2e5a562a14e9c677a16fa7b65"
server_key_iv: "855b92e3e70909d2a51d555fda3ab12d"
server_key_version: "encryptionkey@0"
I20260923 17:31:29.126703 22256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:29.146801 22256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:29.151171 22256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:29.151434 22256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:29.152402 22256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:29.153828 22256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:29.153905 22256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:29.154024 22256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:29.154075 22256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:29.185009 22256 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:39175
I20260923 17:31:29.185053 22383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:39175 every 8 connection(s)
I20260923 17:31:29.186357 22256 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:31:29.194399 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 22256
I20260923 17:31:29.194638 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:31:29.195266 8311 external_mini_cluster.cc:1428] Setting key b6a7e8c8cf8f488b64b64d503cd0514f
Sep 23 17:31:29 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184689, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:29.206410 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:29.188933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:51805 (local address 127.8.29.254:32825)
0923 17:31:29.189330 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:29.189335 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:29.190437 (+ 1102us) server_negotiation.cc:407] Connection header received
0923 17:31:29.192271 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:29.192277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:29.192394 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:29.192502 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:29.194033 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:29.194840 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:29.196508 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:29.196926 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:29.197031 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:29.200386 (+ 3355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:29.200392 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:29.200395 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:29.200427 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:29.203416 (+ 2989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:29.204169 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:29.204175 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:29.204179 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:29.204308 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:29.204708 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:29.204712 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:29.204714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:29.204997 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:29.205127 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:29.205835 (+ 708us) server_negotiation.cc:299] Negotiation successful
0923 17:31:29.206059 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":186,"threads_started":1}
I20260923 17:31:29.207422 22387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:29.189539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:51805)
0923 17:31:29.190274 (+ 735us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:29.190318 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:29.192004 (+ 1686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:29.192748 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:29.192756 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:29.193265 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:29.193839 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:29.193857 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:29.194989 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:29.194993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:29.196365 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:29.196379 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:29.197139 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:29.197146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:29.197357 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:29.197972 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:29.197999 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:29.200156 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:29.203560 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:29.203570 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:29.203587 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:29.204051 (+ 464us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:29.204411 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:29.204416 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:29.204420 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:29.204604 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:29.205117 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:29.205125 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:29.205471 (+ 346us) client_negotiation.cc:769] Sending connection context
0923 17:31:29.205718 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 17:31:29.206264 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":297,"threads_started":1}
I20260923 17:31:29.209059 22384 heartbeater.cc:347] Connected to a master server at 127.8.29.254:32825
I20260923 17:31:29.209352 22384 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:29.210085 22384 heartbeater.cc:511] Master 127.8.29.254:32825 requested a full tablet report, sending...
I20260923 17:31:29.212530 22195 ts_manager.cc:194] Registered new tserver with Master: 6679092b2487475e85075d068cbf7ab2 (127.8.29.193:39175)
I20260923 17:31:29.214684 22195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:51805
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:31:29.245070 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:32825
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:29.565047 22390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:29.565322 22390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:29.565421 22390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:29.572566 22390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:29.572691 22390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:29.572817 22390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:31:29.582769 22390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:29.584061 22390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:29.585351 22390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:29.598050 22395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:29.598145 22396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:29.598141 22398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:29.598850 22390 server_base.cc:1034] running on GCE node
I20260923 17:31:29.599432 22390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:29.600289 22390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:29.601482 22390 hybrid_clock.cc:648] HybridClock initialized: now 1790184689601437 us; error 35 us; skew 500 ppm
I20260923 17:31:29.601765 22390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:29 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184689, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:29.604727 22390 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:31:29.606041 22390 webserver.cc:460] Webserver started at http://127.8.29.194:39189/ using document root <none> and password file <none>
I20260923 17:31:29.606730 22390 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:29.606824 22390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:29.607059 22390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:29.609797 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "3b7142acd7f7424aac4861cb0d52145a"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "0e0c5445eb4e967c3732d69ca0ec608e"
server_key_iv: "c63ab582bba415480d23369be6dd0d33"
server_key_version: "encryptionkey@0"
I20260923 17:31:29.610510 22390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "3b7142acd7f7424aac4861cb0d52145a"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "0e0c5445eb4e967c3732d69ca0ec608e"
server_key_iv: "c63ab582bba415480d23369be6dd0d33"
server_key_version: "encryptionkey@0"
I20260923 17:31:29.615448 22390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:31:29.619037 22405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:29.620357 22390 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:31:29.620532 22390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "3b7142acd7f7424aac4861cb0d52145a"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "0e0c5445eb4e967c3732d69ca0ec608e"
server_key_iv: "c63ab582bba415480d23369be6dd0d33"
server_key_version: "encryptionkey@0"
I20260923 17:31:29.620707 22390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:29.637265 22390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:29.641588 22390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:29.641857 22390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:29.642822 22390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:29.644096 22390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:29.644170 22390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:29.644275 22390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:29.644315 22390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:29.673139 22390 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:44349
I20260923 17:31:29.673187 22517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:44349 every 8 connection(s)
I20260923 17:31:29.674360 22390 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:31:29.675716 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 22390
I20260923 17:31:29.675922 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:31:29.676456 8311 external_mini_cluster.cc:1428] Setting key 24267e6fc164bc561d18fcb68ac64aa4
Sep 23 17:31:29 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184689, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:29.694134 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:29.677106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:35489 (local address 127.8.29.254:32825)
0923 17:31:29.677302 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:29.677307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:29.678592 (+ 1285us) server_negotiation.cc:407] Connection header received
0923 17:31:29.680404 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:29.680410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:29.680493 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:29.680589 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:29.682025 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:29.683029 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:29.684658 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:29.685148 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:29.685264 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:29.688911 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:29.688917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:29.688921 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:29.688954 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:29.691331 (+ 2377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:29.691997 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:29.692001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:29.692003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:29.692133 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:29.692533 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:29.692537 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:29.692539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:29.692796 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:29.692925 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:29.693574 (+ 649us) server_negotiation.cc:299] Negotiation successful
0923 17:31:29.693800 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:31:29.695163 22520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:29.677630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:35489)
0923 17:31:29.678410 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:29.678490 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:29.680139 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:29.680792 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:29.680801 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:29.681287 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:29.681856 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:29.681873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:29.683201 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:29.683205 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:29.684520 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:29.684533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:29.685390 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:29.685397 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:29.685632 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:29.686272 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:29.686300 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:29.688690 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:29.691459 (+ 2769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:29.691468 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:29.691485 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:29.691877 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:29.692231 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:29.692236 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:29.692240 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:29.692422 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:29.692920 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:29.692929 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:29.693244 (+ 315us) client_negotiation.cc:769] Sending connection context
0923 17:31:29.693488 (+ 244us) client_negotiation.cc:240] Negotiation successful
0923 17:31:29.694013 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":316,"threads_started":1}
I20260923 17:31:29.696602 22518 heartbeater.cc:347] Connected to a master server at 127.8.29.254:32825
I20260923 17:31:29.696897 22518 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
I20260923 17:31:29.697731 22518 heartbeater.cc:511] Master 127.8.29.254:32825 requested a full tablet report, sending...
I20260923 17:31:29.699206 22195 ts_manager.cc:194] Registered new tserver with Master: 3b7142acd7f7424aac4861cb0d52145a (127.8.29.194:44349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
I20260923 17:31:29.700076 22195 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:35489
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:31:29.725555 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:32825
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:30.043560 22523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:30.043849 22523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:30.043967 22523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:30.051229 22523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:30.051344 22523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:30.051498 22523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:31:30.061406 22523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:30.062776 22523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:30.064110 22523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:30.077226 22528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:30.077189 22531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:30.077247 22529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:30.077983 22523 server_base.cc:1034] running on GCE node
I20260923 17:31:30.078645 22523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:30.079674 22523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:30.080883 22523 hybrid_clock.cc:648] HybridClock initialized: now 1790184690080830 us; error 49 us; skew 500 ppm
I20260923 17:31:30.081221 22523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:30 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184690, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:30.084345 22523 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:31:30.085822 22523 webserver.cc:460] Webserver started at http://127.8.29.195:37743/ using document root <none> and password file <none>
I20260923 17:31:30.086562 22523 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:30.086670 22523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:30.086939 22523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:30.089872 22523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c"
format_stamp: "Formatted at 2026-09-23 17:31:30 on dist-test-slave-p7jt"
server_key: "1eedaf935529bea2cc320f438b799f93"
server_key_iv: "1006c8aec00fbc00d06003b427baeaf3"
server_key_version: "encryptionkey@0"
I20260923 17:31:30.090684 22523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c"
format_stamp: "Formatted at 2026-09-23 17:31:30 on dist-test-slave-p7jt"
server_key: "1eedaf935529bea2cc320f438b799f93"
server_key_iv: "1006c8aec00fbc00d06003b427baeaf3"
server_key_version: "encryptionkey@0"
I20260923 17:31:30.095922 22523 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:31:30.099891 22538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:30.101514 22523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:31:30.101675 22523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c"
format_stamp: "Formatted at 2026-09-23 17:31:30 on dist-test-slave-p7jt"
server_key: "1eedaf935529bea2cc320f438b799f93"
server_key_iv: "1006c8aec00fbc00d06003b427baeaf3"
server_key_version: "encryptionkey@0"
I20260923 17:31:30.101891 22523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:30.129969 22523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:30.134176 22523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:30.134490 22523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:30.135658 22523 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:30.137212 22523 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:30.137292 22523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:30.137383 22523 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:30.137424 22523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:30.168124 22523 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:40489
I20260923 17:31:30.168192 22650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:40489 every 8 connection(s)
I20260923 17:31:30.169389 22523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:31:30.177500 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 22523
I20260923 17:31:30.177672 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:31:30.178143 8311 external_mini_cluster.cc:1428] Setting key 34c785b97f039488e6182569a153b5b9
Sep 23 17:31:30 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184690, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:30.188733 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.171977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:56271 (local address 127.8.29.254:32825)
0923 17:31:30.172165 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.172169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.173317 (+ 1148us) server_negotiation.cc:407] Connection header received
0923 17:31:30.175213 (+ 1896us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.175219 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.175303 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.175390 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:30.176912 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.177677 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.179341 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.179747 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.179867 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.183309 (+ 3442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:30.183314 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:30.183318 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:30.183352 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:30.185866 (+ 2514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.186572 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.186576 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.186578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.186704 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.187106 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.187109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.187111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.187383 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:30.187520 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.188181 (+ 661us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.188403 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:31:30.189653 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.172433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:56271)
0923 17:31:30.173149 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:30.173194 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:30.174937 (+ 1743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:30.175644 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:30.175653 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:30.176152 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:30.176727 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.176745 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.177820 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.177823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.179212 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.179226 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.179959 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.179967 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.180252 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.180921 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:30.180943 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:30.183107 (+ 2164us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:30.186000 (+ 2893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:30.186010 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:30.186027 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:30.186441 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:30.186805 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:30.186811 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:30.186815 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:30.186992 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:30.187507 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:30.187514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:30.187824 (+ 310us) client_negotiation.cc:769] Sending connection context
0923 17:31:30.188053 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 17:31:30.188576 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":274,"threads_started":1}
I20260923 17:31:30.191045 22651 heartbeater.cc:347] Connected to a master server at 127.8.29.254:32825
I20260923 17:31:30.191367 22651 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:30.192117 22651 heartbeater.cc:511] Master 127.8.29.254:32825 requested a full tablet report, sending...
I20260923 17:31:30.193624 22194 ts_manager.cc:194] Registered new tserver with Master: cee9c4d3dbbe4c2584ed16f408c4b20c (127.8.29.195:40489)
I20260923 17:31:30.194594 22194 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:56271
I20260923 17:31:30.206218 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:31:30.218236 22384 heartbeater.cc:503] Master 127.8.29.254:32825 was elected leader, sending a full tablet report...
I20260923 17:31:30.224996 22386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.210807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46160 (local address 127.8.29.254:32825)
0923 17:31:30.210973 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.210978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.211409 (+ 431us) server_negotiation.cc:407] Connection header received
0923 17:31:30.211548 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.211552 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.211622 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.211719 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:30.213376 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.214163 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.215716 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.216146 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.216253 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.218903 (+ 2650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:30.218908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:30.218912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:30.218950 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:30.222087 (+ 3137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.222956 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.222961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.222964 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.223122 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.223540 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.223544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.223546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.223970 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:30.224165 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.224561 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.224740 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:31:30.229667 22194 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46160:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:31:30.237077 22194 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:31:30.262430 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.249733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40750 (local address 127.8.29.193:39175)
0923 17:31:30.250964 (+ 1231us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.250973 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.252081 (+ 1108us) server_negotiation.cc:407] Connection header received
0923 17:31:30.252497 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.252506 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.252779 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.252944 (+ 165us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:30.253957 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.255555 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.259360 (+ 3805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.260872 (+ 1512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.261001 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.261255 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.261892 (+ 637us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.262078 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":915,"threads_started":1}
I20260923 17:31:30.262641 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.250409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48910 (local address 127.8.29.195:40489)
0923 17:31:30.251315 (+ 906us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.251324 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.252072 (+ 748us) server_negotiation.cc:407] Connection header received
0923 17:31:30.253542 (+ 1470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.253557 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.253820 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.253971 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:30.254941 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.257631 (+ 2690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.260142 (+ 2511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.261545 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.261663 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.261901 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.262123 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.262278 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":734,"thread_start_us":555,"threads_started":1}
I20260923 17:31:30.262693 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.249727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:40489 (local address 127.0.0.1:48910)
0923 17:31:30.251897 (+ 2170us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:30.251956 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:30.252239 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:30.254284 (+ 2045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:30.254289 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:30.254312 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:30.254737 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.254750 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.257843 (+ 3093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.257849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.259987 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.259999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.261750 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.261757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.261935 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.261965 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:31:30.262033 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 17:31:30.262244 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2013,"mutex_wait_us":222,"spinlock_wait_cycles":8192,"thread_start_us":1753,"threads_started":1}
I20260923 17:31:30.264159 22665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.249392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39175 (local address 127.0.0.1:40750)
0923 17:31:30.251897 (+ 2505us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:30.251960 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:30.252239 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:30.253307 (+ 1068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:30.253314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:30.253343 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:30.253760 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.253774 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.255733 (+ 1959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.255739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.258592 (+ 2853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.258610 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.261387 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.261393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.261607 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.261663 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 17:31:30.262926 (+ 1263us) client_negotiation.cc:240] Negotiation successful
0923 17:31:30.263829 (+ 903us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2220,"mutex_wait_us":1,"spinlock_wait_cycles":3328,"thread_start_us":2225,"threads_started":1}
I20260923 17:31:30.267427 22667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.252304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:44349 (local address 127.0.0.1:38314)
0923 17:31:30.259346 (+ 7042us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:30.259365 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:30.259490 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:30.260398 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:30.260402 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:30.260429 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:30.260789 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.260802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.262724 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.262729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.264473 (+ 1744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.264488 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.266724 (+ 2236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.266729 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.266852 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.266875 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:31:30.266955 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 17:31:30.267090 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6916,"thread_start_us":6767,"threads_started":1}
I20260923 17:31:30.266808 22586 tablet_service.cc:1467] Processing CreateTablet for tablet fdcd9b510d6d4e45be52d65a8b2bb09d (DEFAULT_TABLE table=test-table [id=bed068ce5d3441698b3274507cbb9d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:30.267486 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.252571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.8.29.194:44349)
0923 17:31:30.258929 (+ 6358us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.258938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.259473 (+ 535us) server_negotiation.cc:407] Connection header received
0923 17:31:30.259641 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.259649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.259955 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.260166 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:30.261021 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.262552 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.264642 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.266236 (+ 1594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.266377 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.266601 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.267029 (+ 428us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.267188 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6180,"thread_start_us":6053,"threads_started":1}
I20260923 17:31:30.268394 22586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcd9b510d6d4e45be52d65a8b2bb09d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:30.268061 22319 tablet_service.cc:1467] Processing CreateTablet for tablet fdcd9b510d6d4e45be52d65a8b2bb09d (DEFAULT_TABLE table=test-table [id=bed068ce5d3441698b3274507cbb9d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:30.269523 22319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcd9b510d6d4e45be52d65a8b2bb09d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:30.274127 22453 tablet_service.cc:1467] Processing CreateTablet for tablet fdcd9b510d6d4e45be52d65a8b2bb09d (DEFAULT_TABLE table=test-table [id=bed068ce5d3441698b3274507cbb9d91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:30.275626 22453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdcd9b510d6d4e45be52d65a8b2bb09d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:30.283563 22669 tablet_bootstrap.cc:492] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: Bootstrap starting.
I20260923 17:31:30.287593 22669 tablet_bootstrap.cc:654] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:30.288537 22670 tablet_bootstrap.cc:492] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: Bootstrap starting.
I20260923 17:31:30.289996 22671 tablet_bootstrap.cc:492] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: Bootstrap starting.
I20260923 17:31:30.290385 22669 log.cc:824] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:30.292336 22670 tablet_bootstrap.cc:654] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:30.293318 22670 log.cc:824] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:30.293375 22671 tablet_bootstrap.cc:654] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:30.293737 22669 tablet_bootstrap.cc:492] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: No bootstrap required, opened a new log
I20260923 17:31:30.294029 22669 ts_tablet_manager.cc:1397] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 17:31:30.294566 22671 log.cc:824] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:30.296154 22670 tablet_bootstrap.cc:492] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: No bootstrap required, opened a new log
I20260923 17:31:30.296398 22670 ts_tablet_manager.cc:1397] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:31:30.297427 22671 tablet_bootstrap.cc:492] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: No bootstrap required, opened a new log
I20260923 17:31:30.297704 22671 ts_tablet_manager.cc:1397] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 17:31:30.303948 22670 raft_consensus.cc:348] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } }
I20260923 17:31:30.304329 22670 raft_consensus.cc:374] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:30.304435 22670 raft_consensus.cc:729] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6679092b2487475e85075d068cbf7ab2, State: Initialized, Role: FOLLOWER
I20260923 17:31:30.304149 22669 raft_consensus.cc:348] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } }
I20260923 17:31:30.304646 22669 raft_consensus.cc:374] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:30.304739 22669 raft_consensus.cc:729] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee9c4d3dbbe4c2584ed16f408c4b20c, State: Initialized, Role: FOLLOWER
I20260923 17:31:30.305032 22670 consensus_queue.cc:260] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } }
I20260923 17:31:30.305316 22669 consensus_queue.cc:260] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } }
I20260923 17:31:30.307047 22670 ts_tablet_manager.cc:1428] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: Time spent starting tablet: real 0.011s user 0.006s sys 0.006s
I20260923 17:31:30.307917 22651 heartbeater.cc:503] Master 127.8.29.254:32825 was elected leader, sending a full tablet report...
I20260923 17:31:30.307976 22671 raft_consensus.cc:348] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } }
I20260923 17:31:30.308423 22671 raft_consensus.cc:374] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:30.308550 22669 ts_tablet_manager.cc:1428] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: Time spent starting tablet: real 0.014s user 0.012s sys 0.004s
I20260923 17:31:30.308583 22671 raft_consensus.cc:729] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7142acd7f7424aac4861cb0d52145a, State: Initialized, Role: FOLLOWER
I20260923 17:31:30.309127 22671 consensus_queue.cc:260] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } }
I20260923 17:31:30.310220 22518 heartbeater.cc:503] Master 127.8.29.254:32825 was elected leader, sending a full tablet report...
I20260923 17:31:30.310842 22671 ts_tablet_manager.cc:1428] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: Time spent starting tablet: real 0.013s user 0.010s sys 0.005s
W20260923 17:31:30.420987 22652 tablet.cc:2369] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:31:30.427342 22519 tablet.cc:2369] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:31:30.438750 22385 tablet.cc:2369] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:31:30.561470 22676 raft_consensus.cc:482] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:31:30.561751 22676 raft_consensus.cc:504] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } }
I20260923 17:31:30.563442 22676 leader_election.cc:288] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6679092b2487475e85075d068cbf7ab2 (127.8.29.193:39175), 3b7142acd7f7424aac4861cb0d52145a (127.8.29.194:44349)
I20260923 17:31:30.569612 22653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.563829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39175 (local address 127.8.29.195:41035)
0923 17:31:30.564039 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:30.564059 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:30.564190 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:30.564561 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:30.564566 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:30.564595 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:30.564802 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.564814 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.566018 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.566022 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.567765 (+ 1743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.567781 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.569059 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.569063 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.569198 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.569222 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:31:30.569297 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 17:31:30.569400 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":40}
I20260923 17:31:30.569728 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.563957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:41035 (local address 127.8.29.193:39175)
0923 17:31:30.564146 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.564151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.564171 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:31:30.564333 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.564339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.564416 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.564510 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:30.564980 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.565877 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.567956 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.568896 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.569060 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.569137 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.569341 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.569483 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:31:30.570626 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.564457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:51287 (local address 127.8.29.194:44349)
0923 17:31:30.564669 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.564676 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.565188 (+ 512us) server_negotiation.cc:407] Connection header received
0923 17:31:30.565308 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.565312 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.565396 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.565486 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:30.565941 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.566688 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.568645 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.569755 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.569904 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.569995 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.570323 (+ 328us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.570420 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:31:30.570793 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.564276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:44349 (local address 127.8.29.195:51287)
0923 17:31:30.565095 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:30.565109 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:30.565202 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:30.565502 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:30.565505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:30.565522 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:30.565801 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.565809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.566830 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.566836 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.568493 (+ 1657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:30.568505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.570043 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.570050 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:30.570167 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.570191 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:31:30.570287 (+ 96us) client_negotiation.cc:240] Negotiation successful
0923 17:31:30.570422 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"mutex_wait_us":1,"thread_start_us":492,"threads_started":1}
I20260923 17:31:30.570796 22339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcd9b510d6d4e45be52d65a8b2bb09d" candidate_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6679092b2487475e85075d068cbf7ab2" is_pre_election: true
I20260923 17:31:30.571348 22339 raft_consensus.cc:2454] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee9c4d3dbbe4c2584ed16f408c4b20c in term 0.
I20260923 17:31:30.571671 22473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcd9b510d6d4e45be52d65a8b2bb09d" candidate_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7142acd7f7424aac4861cb0d52145a" is_pre_election: true
I20260923 17:31:30.571987 22539 leader_election.cc:302] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6679092b2487475e85075d068cbf7ab2, cee9c4d3dbbe4c2584ed16f408c4b20c; no voters:
I20260923 17:31:30.572187 22473 raft_consensus.cc:2454] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee9c4d3dbbe4c2584ed16f408c4b20c in term 0.
I20260923 17:31:30.572355 22676 raft_consensus.cc:2781] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:31:30.572507 22676 raft_consensus.cc:482] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:31:30.572610 22676 raft_consensus.cc:3037] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:30.574905 22676 raft_consensus.cc:504] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } }
I20260923 17:31:30.575485 22676 leader_election.cc:288] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [CANDIDATE]: Term 1 election: Requested vote from peers 6679092b2487475e85075d068cbf7ab2 (127.8.29.193:39175), 3b7142acd7f7424aac4861cb0d52145a (127.8.29.194:44349)
I20260923 17:31:30.575894 22339 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcd9b510d6d4e45be52d65a8b2bb09d" candidate_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6679092b2487475e85075d068cbf7ab2"
I20260923 17:31:30.575987 22473 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdcd9b510d6d4e45be52d65a8b2bb09d" candidate_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7142acd7f7424aac4861cb0d52145a"
I20260923 17:31:30.576076 22339 raft_consensus.cc:3037] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:30.576177 22473 raft_consensus.cc:3037] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:30.578423 22339 raft_consensus.cc:2454] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee9c4d3dbbe4c2584ed16f408c4b20c in term 1.
I20260923 17:31:30.578869 22539 leader_election.cc:302] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6679092b2487475e85075d068cbf7ab2, cee9c4d3dbbe4c2584ed16f408c4b20c; no voters:
I20260923 17:31:30.579118 22676 raft_consensus.cc:2781] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:31:30.579420 22473 raft_consensus.cc:2454] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee9c4d3dbbe4c2584ed16f408c4b20c in term 1.
I20260923 17:31:30.579650 22676 raft_consensus.cc:686] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 LEADER]: Becoming Leader. State: Replica: cee9c4d3dbbe4c2584ed16f408c4b20c, State: Running, Role: LEADER
I20260923 17:31:30.580188 22676 consensus_queue.cc:237] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } }
I20260923 17:31:30.584977 22193 catalog_manager.cc:5495] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c reported cstate change: term changed from 0 to 1, leader changed from <none> to cee9c4d3dbbe4c2584ed16f408c4b20c (127.8.29.195). New cstate: current_term: 1 leader_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } health_report { overall_health: UNKNOWN } } }
I20260923 17:31:30.628384 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.622883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48922 (local address 127.8.29.195:40489)
0923 17:31:30.623051 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.623056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.623074 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:31:30.623160 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.623163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.623291 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.623463 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:31:30.623912 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.624677 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.626181 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.626642 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.626793 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.627293 (+ 500us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:31:30.627816 (+ 523us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:31:30.627951 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.628050 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.628135 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:31:30.637151 22473 raft_consensus.cc:1261] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [term 1 FOLLOWER]: Refusing update from remote peer cee9c4d3dbbe4c2584ed16f408c4b20c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:30.637151 22339 raft_consensus.cc:1261] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [term 1 FOLLOWER]: Refusing update from remote peer cee9c4d3dbbe4c2584ed16f408c4b20c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:30.638079 22679 consensus_queue.cc:1035] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:30.638442 22676 consensus_queue.cc:1035] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:30.663583 22193 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:31:30.663902 22193 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46160:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:31:30.667816 22193 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Sending DeleteTablet for 3 replicas of tablet fdcd9b510d6d4e45be52d65a8b2bb09d
I20260923 17:31:30.669073 22586 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcd9b510d6d4e45be52d65a8b2bb09d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:30 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:48910
I20260923 17:31:30.669176 22453 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcd9b510d6d4e45be52d65a8b2bb09d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:30 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:38314
I20260923 17:31:30.669478 22319 tablet_service.cc:1514] Processing DeleteTablet for tablet fdcd9b510d6d4e45be52d65a8b2bb09d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:30 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:40750
I20260923 17:31:30.670320 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 22256
I20260923 17:31:30.680253 22694 tablet_replica.cc:330] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: stopping tablet replica
I20260923 17:31:30.681484 22694 raft_consensus.cc:2229] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:31:30.682770 22694 raft_consensus.cc:2258] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:30.682926 22692 tablet_replica.cc:330] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: stopping tablet replica
I20260923 17:31:30.683305 22692 raft_consensus.cc:2229] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:31:30.683869 22692 raft_consensus.cc:2258] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:30.686527 22694 ts_tablet_manager.cc:1905] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:30.688217 22693 tablet_replica.cc:330] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: stopping tablet replica
I20260923 17:31:30.688621 22693 raft_consensus.cc:2229] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:31:30.689146 22693 raft_consensus.cc:2258] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:30.691394 22692 ts_tablet_manager.cc:1905] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:30.692971 22693 ts_tablet_manager.cc:1905] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:30.696496 22694 ts_tablet_manager.cc:1918] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:30.696736 22694 log.cc:1196] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/fdcd9b510d6d4e45be52d65a8b2bb09d
I20260923 17:31:30.697809 22694 ts_tablet_manager.cc:1939] T fdcd9b510d6d4e45be52d65a8b2bb09d P 6679092b2487475e85075d068cbf7ab2: Deleting consensus metadata
I20260923 17:31:30.698252 22692 ts_tablet_manager.cc:1918] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:30.698446 22692 log.cc:1196] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/fdcd9b510d6d4e45be52d65a8b2bb09d
I20260923 17:31:30.698987 22692 ts_tablet_manager.cc:1939] T fdcd9b510d6d4e45be52d65a8b2bb09d P cee9c4d3dbbe4c2584ed16f408c4b20c: Deleting consensus metadata
I20260923 17:31:30.699621 22693 ts_tablet_manager.cc:1918] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:30.699783 22693 log.cc:1196] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/fdcd9b510d6d4e45be52d65a8b2bb09d
I20260923 17:31:30.700296 22693 ts_tablet_manager.cc:1939] T fdcd9b510d6d4e45be52d65a8b2bb09d P 3b7142acd7f7424aac4861cb0d52145a: Deleting consensus metadata
I20260923 17:31:30.700873 22179 catalog_manager.cc:4841] TS 6679092b2487475e85075d068cbf7ab2 (127.8.29.193:39175): tablet fdcd9b510d6d4e45be52d65a8b2bb09d (table test-table [id=bed068ce5d3441698b3274507cbb9d91]) successfully deleted
I20260923 17:31:30.700897 22182 catalog_manager.cc:4841] TS cee9c4d3dbbe4c2584ed16f408c4b20c (127.8.29.195:40489): tablet fdcd9b510d6d4e45be52d65a8b2bb09d (table test-table [id=bed068ce5d3441698b3274507cbb9d91]) successfully deleted
I20260923 17:31:30.701915 22179 catalog_manager.cc:4841] TS 3b7142acd7f7424aac4861cb0d52145a (127.8.29.194:44349): tablet fdcd9b510d6d4e45be52d65a8b2bb09d (table test-table [id=bed068ce5d3441698b3274507cbb9d91]) successfully deleted
Sep 23 17:31:30 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:31:30.710785 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.671559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40758 (local address 127.8.29.193:39175)
0923 17:31:30.671742 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.671749 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.691848 (+ 20099us) server_negotiation.cc:407] Connection header received
0923 17:31:30.692159 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.692165 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.692249 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.692340 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:30.694310 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.695437 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.697387 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.698065 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.698170 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.703784 (+ 5614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:30.703800 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:30.703819 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:30.703863 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:30.707550 (+ 3687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.708522 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.708530 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.708532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.708664 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.709144 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.709149 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.709152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.709588 (+ 436us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:30.709907 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.710234 (+ 327us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.710443 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:31:30.711498 22379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:30.802774 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 22256
I20260923 17:31:30.824900 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 22390
Sep 23 17:31:30 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:31:30.841951 22668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.825739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38328 (local address 127.8.29.194:44349)
0923 17:31:30.825901 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.825906 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.825933 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:31:30.826109 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.826113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.826189 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.826267 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:30.827907 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.828593 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.830500 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.830885 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.831004 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.836256 (+ 5252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:30.836273 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:30.836289 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:30.836326 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:30.839105 (+ 2779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.839943 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.839951 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.839953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.840079 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.840467 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.840470 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.840472 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.840863 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:30.841137 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.841458 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.841632 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:31:30.842656 22513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:30.932226 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 22390
I20260923 17:31:30.947173 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 22523
Sep 23 17:31:30 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:31:30.966217 22666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:30.948010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48924 (local address 127.8.29.195:40489)
0923 17:31:30.948184 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:30.948188 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:30.948253 (+ 65us) server_negotiation.cc:407] Connection header received
0923 17:31:30.948456 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:30.948459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:30.948538 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:30.948619 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:30.950423 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.951319 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.953248 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:30.953723 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:30.953821 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:30.959693 (+ 5872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:30.959710 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:30.959729 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:30.959781 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:30.962928 (+ 3147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.963949 (+ 1021us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.963958 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.963961 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.964139 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:30.964612 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:30.964617 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:30.964619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:30.965027 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:30.965286 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:30.965682 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 17:31:30.965861 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:31:30.966954 22646 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:31.088660 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 22523
I20260923 17:31:31.105244 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 22163
I20260923 17:31:31.106449 22225 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:31.217281 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 22163
I20260923 17:31:31.237353 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:32825
--webserver_interface=127.8.29.254
--webserver_port=46249
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:31.561597 22700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:31.561940 22700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:31.562077 22700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:31.569603 22700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:31.569758 22700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:31.569839 22700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:31.569924 22700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:31.569986 22700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:31.580402 22700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:32825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=46249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:31.581763 22700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:31.583168 22700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:31.607529 22707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:31.607550 22709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:31.607570 22706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:31.608826 22700 server_base.cc:1034] running on GCE node
I20260923 17:31:31.609712 22700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:31.611428 22700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:31.612658 22700 hybrid_clock.cc:648] HybridClock initialized: now 1790184691612594 us; error 52 us; skew 500 ppm
I20260923 17:31:31.613081 22700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:31 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184691, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:31.616727 22700 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:31:31.618556 22700 webserver.cc:460] Webserver started at http://127.8.29.254:46249/ using document root <none> and password file <none>
I20260923 17:31:31.619375 22700 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:31.619524 22700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:31.625877 22700 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.001s
I20260923 17:31:31.628997 22716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:31.630451 22700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260923 17:31:31.630661 22700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "2efa957b67ab41ecb8822193aa2053df"
format_stamp: "Formatted at 2026-09-23 17:31:28 on dist-test-slave-p7jt"
server_key: "b14cf4a70ec6711b503d8391ee4130a6"
server_key_iv: "226d5857b1fcba3fed365cebb92fbdee"
server_key_version: "encryptionkey@0"
I20260923 17:31:31.631759 22700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:31.658624 22700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:31.663389 22700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:31.663676 22700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:31.681510 22700 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:32825
I20260923 17:31:31.681530 22767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:32825 every 8 connection(s)
I20260923 17:31:31.683092 22700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:31:31.689728 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 22700
I20260923 17:31:31.690429 22768 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:31:31.690768 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:39175
--local_ip_for_outbound_sockets=127.8.29.193
--tserver_master_addrs=127.8.29.254:32825
--webserver_port=43429
--webserver_interface=127.8.29.193
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:31:31.695106 22768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Bootstrap starting.
I20260923 17:31:31.715981 22768 log.cc:824] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:31.728653 22768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:31:31.729228 22768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Bootstrap complete.
I20260923 17:31:31.738708 22768 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:31.739544 22768 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efa957b67ab41ecb8822193aa2053df, State: Initialized, Role: FOLLOWER
I20260923 17:31:31.740155 22768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:31.740371 22768 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:31:31.740494 22768 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:31:31.740645 22768 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:31:31.744428 22768 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:31.744926 22768 leader_election.cc:302] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2efa957b67ab41ecb8822193aa2053df; no voters:
I20260923 17:31:31.745504 22768 leader_election.cc:288] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:31:31.745643 22772 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:31:31.746738 22772 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 2 LEADER]: Becoming Leader. State: Replica: 2efa957b67ab41ecb8822193aa2053df, State: Running, Role: LEADER
I20260923 17:31:31.747360 22772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:31.747737 22768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:31:31.749739 22774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2efa957b67ab41ecb8822193aa2053df. Latest consensus state: current_term: 2 leader_uuid: "2efa957b67ab41ecb8822193aa2053df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } } }
I20260923 17:31:31.750099 22774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:31.750672 22773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2efa957b67ab41ecb8822193aa2053df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } } }
I20260923 17:31:31.750959 22773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:31.761199 22778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:31:31.775604 22778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:31:31.777387 22778 catalog_manager.cc:1259] Loaded cluster ID: 8599245691b64b1c8c5541508ae482a5
I20260923 17:31:31.777586 22778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:31:31.781196 22778 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 17:31:31.788501 22790 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 17:31:31.795567 22778 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 17:31:31.803529 22778 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:31:31.805307 22778 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Loaded TSK: 0
I20260923 17:31:31.806912 22778 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:31:32.038277 22770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:32.038666 22770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:32.038803 22770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:32.046020 22770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:32.046142 22770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:32.046295 22770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:31:32.056237 22770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:39175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=43429
--tserver_master_addrs=127.8.29.254:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:32.057580 22770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:32.058979 22770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:32.074178 22796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:32.074206 22797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:32.074687 22799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:32.075305 22770 server_base.cc:1034] running on GCE node
I20260923 17:31:32.075995 22770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:32.077029 22770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:32.078279 22770 hybrid_clock.cc:648] HybridClock initialized: now 1790184692078218 us; error 46 us; skew 500 ppm
I20260923 17:31:32.078670 22770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:32 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184692, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:32.081928 22770 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:31:32.083535 22770 webserver.cc:460] Webserver started at http://127.8.29.193:43429/ using document root <none> and password file <none>
I20260923 17:31:32.084250 22770 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:32.084372 22770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:32.089895 22770 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:31:32.092700 22806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:32.094393 22770 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:31:32.094599 22770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "6679092b2487475e85075d068cbf7ab2"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "9c8dc2e2e5a562a14e9c677a16fa7b65"
server_key_iv: "855b92e3e70909d2a51d555fda3ab12d"
server_key_version: "encryptionkey@0"
I20260923 17:31:32.095572 22770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:32.108984 22770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:32.113180 22770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:32.113436 22770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:32.114426 22770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:32.115895 22770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:32.116000 22770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:32.116101 22770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:32.116168 22770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:32.146656 22770 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:39175
I20260923 17:31:32.146705 22918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:39175 every 8 connection(s)
I20260923 17:31:32.148162 22770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:31:32.155879 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 22770
I20260923 17:31:32.156528 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:44349
--local_ip_for_outbound_sockets=127.8.29.194
--tserver_master_addrs=127.8.29.254:32825
--webserver_port=39189
--webserver_interface=127.8.29.194
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:31:32 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184692, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:32.171736 22922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:32.151248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:46249)
0923 17:31:32.151960 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:32.151998 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:32.153677 (+ 1679us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:32.155102 (+ 1425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:32.155113 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:32.155759 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:32.156410 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:32.156429 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:32.158090 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:32.158094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:32.159375 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:32.159396 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:32.160451 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:32.160458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:32.160709 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:32.161404 (+ 695us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:32.161427 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:32.163697 (+ 2270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:32.167184 (+ 3487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:32.167194 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:32.167211 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:32.167663 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:32.168155 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:32.168160 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:32.168164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:32.168360 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:32.169139 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:32.169148 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:32.169510 (+ 362us) client_negotiation.cc:769] Sending connection context
0923 17:31:32.169757 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 17:31:32.170345 (+ 588us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":297,"threads_started":1}
I20260923 17:31:32.171736 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:32.151009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:46249 (local address 127.8.29.254:32825)
0923 17:31:32.151588 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:32.151606 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:32.152133 (+ 527us) server_negotiation.cc:407] Connection header received
0923 17:31:32.154191 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:32.154206 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:32.154591 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:32.154938 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:32.156629 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:32.157933 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:32.159530 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:32.160204 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:32.160405 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:32.163980 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:32.163988 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:32.164004 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:32.164042 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:32.167032 (+ 2990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:32.167812 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:32.167819 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:32.167825 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:32.168035 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:32.168486 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:32.168490 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:32.168492 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:32.168994 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:32.169187 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:32.170018 (+ 831us) server_negotiation.cc:299] Negotiation successful
0923 17:31:32.170437 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":231,"threads_started":1}
I20260923 17:31:32.175083 22919 heartbeater.cc:347] Connected to a master server at 127.8.29.254:32825
I20260923 17:31:32.175372 22919 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:32.176119 22919 heartbeater.cc:511] Master 127.8.29.254:32825 requested a full tablet report, sending...
I20260923 17:31:32.178558 22733 ts_manager.cc:194] Registered new tserver with Master: 6679092b2487475e85075d068cbf7ab2 (127.8.29.193:39175)
I20260923 17:31:32.180860 22733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:46249
W20260923 17:31:32.479501 22923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:32.479792 22923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:32.479887 22923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:32.487128 22923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:32.487241 22923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:32.487351 22923 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:31:32.497222 22923 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:44349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=39189
--tserver_master_addrs=127.8.29.254:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:32.498499 22923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:32.499886 22923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:32.512496 22928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:32.512609 22929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:32.512573 22931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:32.513598 22923 server_base.cc:1034] running on GCE node
I20260923 17:31:32.514195 22923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:32.515206 22923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:32.516435 22923 hybrid_clock.cc:648] HybridClock initialized: now 1790184692516368 us; error 62 us; skew 500 ppm
I20260923 17:31:32.516767 22923 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:32 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184692, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:32.519912 22923 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:31:32.523340 22923 webserver.cc:460] Webserver started at http://127.8.29.194:39189/ using document root <none> and password file <none>
I20260923 17:31:32.524294 22923 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:32.524451 22923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:32.531056 22923 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:31:32.534242 22938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:32.535921 22923 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:31:32.536103 22923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "3b7142acd7f7424aac4861cb0d52145a"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "0e0c5445eb4e967c3732d69ca0ec608e"
server_key_iv: "c63ab582bba415480d23369be6dd0d33"
server_key_version: "encryptionkey@0"
I20260923 17:31:32.537135 22923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:32.575243 22923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:32.579867 22923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:32.580127 22923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:32.581118 22923 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:32.582494 22923 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:32.582593 22923 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:32.582702 22923 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:32.582778 22923 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:32.615795 22923 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:44349
I20260923 17:31:32.615854 23050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:44349 every 8 connection(s)
I20260923 17:31:32.617487 22923 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:31:32.619220 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 22923
I20260923 17:31:32.619937 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:40489
--local_ip_for_outbound_sockets=127.8.29.195
--tserver_master_addrs=127.8.29.254:32825
--webserver_port=37743
--webserver_interface=127.8.29.195
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:31:32 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184692, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:32.653097 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:32.630492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:36745 (local address 127.8.29.254:32825)
0923 17:31:32.630795 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:32.630802 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:32.632241 (+ 1439us) server_negotiation.cc:407] Connection header received
0923 17:31:32.634895 (+ 2654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:32.634901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:32.634991 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:32.635060 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:32.637420 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:32.638407 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:32.641029 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:32.641719 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:32.641854 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:32.646536 (+ 4682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:32.646542 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:32.646548 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:32.646594 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:32.649882 (+ 3288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:32.650706 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:32.650712 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:32.650716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:32.650874 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:32.651283 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:32.651288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:32.651291 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:32.651643 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:32.651788 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:32.652504 (+ 716us) server_negotiation.cc:299] Negotiation successful
0923 17:31:32.652750 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:31:32.653848 23054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:32.631084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:36745)
0923 17:31:32.632078 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:32.632134 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:32.634512 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:32.635643 (+ 1131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:32.635655 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:32.636424 (+ 769us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:32.637200 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:32.637224 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:32.638644 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:32.638650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:32.640847 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:32.640865 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:32.641968 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:32.641977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:32.642312 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:32.643091 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:32.643126 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:32.646211 (+ 3085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:32.650028 (+ 3817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:32.650036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:32.650052 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:32.650585 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:32.650970 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:32.650973 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:32.650975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:32.651178 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:32.651743 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:32.651749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:32.652095 (+ 346us) client_negotiation.cc:769] Sending connection context
0923 17:31:32.652327 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 17:31:32.652743 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":397,"threads_started":1}
I20260923 17:31:32.655352 23051 heartbeater.cc:347] Connected to a master server at 127.8.29.254:32825
I20260923 17:31:32.655691 23051 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:32.656530 23051 heartbeater.cc:511] Master 127.8.29.254:32825 requested a full tablet report, sending...
I20260923 17:31:32.658080 22733 ts_manager.cc:194] Registered new tserver with Master: 3b7142acd7f7424aac4861cb0d52145a (127.8.29.194:44349)
I20260923 17:31:32.658922 22733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:36745
W20260923 17:31:32.949220 23053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:32.949529 23053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:32.949671 23053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:32.956884 23053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:32.957000 23053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:32.957146 23053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:31:32.967247 23053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:40489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=37743
--tserver_master_addrs=127.8.29.254:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:32.968549 23053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:32.969849 23053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:32.982611 23059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:32.982649 23060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:32.982681 23062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:32.983494 23053 server_base.cc:1034] running on GCE node
I20260923 17:31:32.984042 23053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:32.984973 23053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:32.986166 23053 hybrid_clock.cc:648] HybridClock initialized: now 1790184692986119 us; error 34 us; skew 500 ppm
I20260923 17:31:32.986562 23053 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:32 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184692, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:32.989944 23053 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:31:32.991545 23053 webserver.cc:460] Webserver started at http://127.8.29.195:37743/ using document root <none> and password file <none>
I20260923 17:31:32.992268 23053 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:32.992408 23053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:32.998338 23053 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260923 17:31:33.001456 23069 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:33.003013 23053 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:31:33.003171 23053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c"
format_stamp: "Formatted at 2026-09-23 17:31:30 on dist-test-slave-p7jt"
server_key: "1eedaf935529bea2cc320f438b799f93"
server_key_iv: "1006c8aec00fbc00d06003b427baeaf3"
server_key_version: "encryptionkey@0"
I20260923 17:31:33.004308 23053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:33.034914 23053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:33.039332 23053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:33.039592 23053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:33.040580 23053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:33.042006 23053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:33.042116 23053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:33.042224 23053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:33.042290 23053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:33.070714 23053 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:40489
I20260923 17:31:33.070761 23181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:40489 every 8 connection(s)
I20260923 17:31:33.072145 23053 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:31:33.078038 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 23053
Sep 23 17:31:33 dist-test-slave-p7jt krb5kdc[22150](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184692, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:33.095121 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.079021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46166 (local address 127.8.29.254:32825)
0923 17:31:33.080117 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.080122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.080139 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:31:33.080179 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.080182 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.080242 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.080318 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:33.082079 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.083193 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.084937 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.085444 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.085539 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.088600 (+ 3061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:33.088606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:33.088610 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:33.088656 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:33.092395 (+ 3739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.093201 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.093205 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.093207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.093359 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.093732 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.093735 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.093737 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.094077 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:33.094315 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.094615 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.094789 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":986,"thread_start_us":837,"threads_started":1}
I20260923 17:31:33.095999 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.074843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:52519 (local address 127.8.29.254:32825)
0923 17:31:33.075045 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.075050 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.076244 (+ 1194us) server_negotiation.cc:407] Connection header received
0923 17:31:33.078067 (+ 1823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.078074 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.078164 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.078276 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:33.080138 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.081265 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.083355 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.084020 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.084159 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.088364 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:33.088370 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:33.088373 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:33.088407 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:33.092395 (+ 3988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.093344 (+ 949us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.093348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.093351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.093505 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.093989 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.093993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.093996 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.094277 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:33.094386 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.095405 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.095624 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":65}
I20260923 17:31:33.097231 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.075380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:52519)
0923 17:31:33.076083 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:33.076121 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:33.077771 (+ 1650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:33.078554 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:33.078562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:33.079122 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:33.079951 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.079978 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.081444 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.081448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.083193 (+ 1745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.083207 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.084265 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.084271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.084503 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.085285 (+ 782us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:33.085309 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:33.088097 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:33.092565 (+ 4468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:33.092574 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:33.092594 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:33.093231 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:33.093607 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:33.093611 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:33.093615 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:33.093867 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:33.094400 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:33.094409 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:33.094941 (+ 532us) client_negotiation.cc:769] Sending connection context
0923 17:31:33.095248 (+ 307us) client_negotiation.cc:240] Negotiation successful
0923 17:31:33.095852 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":275,"threads_started":1}
I20260923 17:31:33.098626 23182 heartbeater.cc:347] Connected to a master server at 127.8.29.254:32825
I20260923 17:31:33.098943 23182 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:33.099711 23182 heartbeater.cc:511] Master 127.8.29.254:32825 requested a full tablet report, sending...
I20260923 17:31:33.101220 22733 ts_manager.cc:194] Registered new tserver with Master: cee9c4d3dbbe4c2584ed16f408c4b20c (127.8.29.195:40489)
I20260923 17:31:33.102038 22733 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:52519
I20260923 17:31:33.110492 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:31:33.116825 22921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.111331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46180 (local address 127.8.29.254:32825)
0923 17:31:33.111476 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.111480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.111908 (+ 428us) server_negotiation.cc:407] Connection header received
0923 17:31:33.112072 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.112075 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.112208 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.112333 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:31:33.112698 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.113414 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.114927 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.115341 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.115437 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.115895 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:31:33.116393 (+ 498us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:31:33.116499 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.116556 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.116623 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260923 17:31:33.117743 22733 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46180:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:31:33.125223 22733 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:31:33.162092 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.137681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48938 (local address 127.8.29.195:40489)
0923 17:31:33.152675 (+ 14994us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.152682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.152713 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:31:33.152814 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.152821 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.153019 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.153202 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:33.155083 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.156705 (+ 1622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.159640 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.160881 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.161006 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.161263 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.161567 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.161735 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14833,"thread_start_us":14659,"threads_started":1}
I20260923 17:31:33.162137 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.137411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:40489 (local address 127.0.0.1:48938)
0923 17:31:33.150025 (+ 12614us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:33.150091 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:33.150331 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:33.153454 (+ 3123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:33.153462 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:33.153482 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:33.154836 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.154850 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.156846 (+ 1996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.156852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.159437 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.159453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.161103 (+ 1650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.161109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.161306 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.161359 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 17:31:33.161476 (+ 117us) client_negotiation.cc:240] Negotiation successful
0923 17:31:33.161716 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12326,"mutex_wait_us":946,"thread_start_us":14415,"threads_started":1}
I20260923 17:31:33.164194 23196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.151642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39175 (local address 127.0.0.1:40774)
0923 17:31:33.156474 (+ 4832us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:33.156493 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:33.156622 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:33.157159 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:33.157164 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:33.157188 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:33.157572 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.157583 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.159507 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.159512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.161412 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.161428 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.163297 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.163303 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.163502 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.163534 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:31:33.163628 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 17:31:33.163774 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4683,"mutex_wait_us":1,"thread_start_us":4540,"threads_started":1}
I20260923 17:31:33.164690 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.138157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.8.29.194:44349)
0923 17:31:33.149223 (+ 11066us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.149234 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.152354 (+ 3120us) server_negotiation.cc:407] Connection header received
0923 17:31:33.152489 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.152497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.152769 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.152952 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:33.153964 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.157354 (+ 3390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.160406 (+ 3052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.163486 (+ 3080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.163633 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.163839 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.164085 (+ 246us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.164250 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10878,"thread_start_us":10744,"threads_started":1}
I20260923 17:31:33.164964 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.137550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:44349 (local address 127.0.0.1:38330)
0923 17:31:33.152071 (+ 14521us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:33.152088 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:33.152200 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:33.153279 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:33.153287 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:33.153320 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:33.153749 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.153770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.157506 (+ 3736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.157511 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.160202 (+ 2691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.160217 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.163730 (+ 3513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.163737 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.163913 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.163944 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 17:31:33.164023 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 17:31:33.164162 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14406,"mutex_wait_us":27,"thread_start_us":11913,"threads_started":1}
I20260923 17:31:33.164968 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.151926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.8.29.193:39175)
0923 17:31:33.156151 (+ 4225us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.156160 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.156607 (+ 447us) server_negotiation.cc:407] Connection header received
0923 17:31:33.156749 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.156768 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.156976 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.157143 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:33.157790 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.159364 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.161623 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.163051 (+ 1428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.163192 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.163439 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.164492 (+ 1053us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.164673 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4058,"thread_start_us":3903,"threads_started":1}
I20260923 17:31:33.166857 23117 tablet_service.cc:1467] Processing CreateTablet for tablet 39be83b677f44e209d20cb1033a01595 (DEFAULT_TABLE table=test-table [id=183a82f9f51a45da95f1eb7ce894a55e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:33.168449 23117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39be83b677f44e209d20cb1033a01595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:33.168190 22986 tablet_service.cc:1467] Processing CreateTablet for tablet 39be83b677f44e209d20cb1033a01595 (DEFAULT_TABLE table=test-table [id=183a82f9f51a45da95f1eb7ce894a55e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:33.168603 22854 tablet_service.cc:1467] Processing CreateTablet for tablet 39be83b677f44e209d20cb1033a01595 (DEFAULT_TABLE table=test-table [id=183a82f9f51a45da95f1eb7ce894a55e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:33.169356 22986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39be83b677f44e209d20cb1033a01595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:33.169948 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 39be83b677f44e209d20cb1033a01595. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:33.181363 23198 tablet_bootstrap.cc:492] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: Bootstrap starting.
I20260923 17:31:33.184280 22919 heartbeater.cc:503] Master 127.8.29.254:32825 was elected leader, sending a full tablet report...
I20260923 17:31:33.186165 23198 tablet_bootstrap.cc:654] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:33.187623 23198 log.cc:824] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:33.189138 23199 tablet_bootstrap.cc:492] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: Bootstrap starting.
I20260923 17:31:33.189889 23200 tablet_bootstrap.cc:492] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: Bootstrap starting.
I20260923 17:31:33.191382 23198 tablet_bootstrap.cc:492] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: No bootstrap required, opened a new log
I20260923 17:31:33.191680 23198 ts_tablet_manager.cc:1397] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 17:31:33.193315 23199 tablet_bootstrap.cc:654] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:33.193487 23200 tablet_bootstrap.cc:654] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:33.194694 23199 log.cc:824] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:33.194732 23200 log.cc:824] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:33.197600 23200 tablet_bootstrap.cc:492] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: No bootstrap required, opened a new log
I20260923 17:31:33.197743 23199 tablet_bootstrap.cc:492] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: No bootstrap required, opened a new log
I20260923 17:31:33.197871 23200 ts_tablet_manager.cc:1397] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 17:31:33.198015 23199 ts_tablet_manager.cc:1397] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260923 17:31:33.201741 23198 raft_consensus.cc:348] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } }
I20260923 17:31:33.202235 23198 raft_consensus.cc:374] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:33.202401 23198 raft_consensus.cc:729] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b7142acd7f7424aac4861cb0d52145a, State: Initialized, Role: FOLLOWER
I20260923 17:31:33.203126 23198 consensus_queue.cc:260] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } }
I20260923 17:31:33.206303 23200 raft_consensus.cc:348] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } }
I20260923 17:31:33.206874 23200 raft_consensus.cc:374] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:33.207018 23200 raft_consensus.cc:729] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6679092b2487475e85075d068cbf7ab2, State: Initialized, Role: FOLLOWER
I20260923 17:31:33.207698 23200 consensus_queue.cc:260] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } }
I20260923 17:31:33.208875 23199 raft_consensus.cc:348] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } }
I20260923 17:31:33.209323 23199 raft_consensus.cc:374] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:33.209422 23199 raft_consensus.cc:729] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee9c4d3dbbe4c2584ed16f408c4b20c, State: Initialized, Role: FOLLOWER
I20260923 17:31:33.209733 23200 ts_tablet_manager.cc:1428] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: Time spent starting tablet: real 0.012s user 0.012s sys 0.001s
I20260923 17:31:33.210393 23199 consensus_queue.cc:260] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } }
I20260923 17:31:33.211686 23182 heartbeater.cc:503] Master 127.8.29.254:32825 was elected leader, sending a full tablet report...
I20260923 17:31:33.212438 23199 ts_tablet_manager.cc:1428] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: Time spent starting tablet: real 0.014s user 0.014s sys 0.001s
I20260923 17:31:33.212962 23051 heartbeater.cc:503] Master 127.8.29.254:32825 was elected leader, sending a full tablet report...
I20260923 17:31:33.213091 23198 ts_tablet_manager.cc:1428] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: Time spent starting tablet: real 0.021s user 0.016s sys 0.003s
W20260923 17:31:33.323732 23183 tablet.cc:2369] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:31:33.369602 23052 tablet.cc:2369] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:31:33.378158 23206 raft_consensus.cc:482] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:31:33.378490 23206 raft_consensus.cc:504] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } }
I20260923 17:31:33.380141 23206 leader_election.cc:288] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b7142acd7f7424aac4861cb0d52145a (127.8.29.194:44349), 6679092b2487475e85075d068cbf7ab2 (127.8.29.193:39175)
I20260923 17:31:33.386379 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.380503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:44349 (local address 127.8.29.195:40891)
0923 17:31:33.380690 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:33.380709 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:33.380849 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:33.381386 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:33.381390 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:33.381422 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:33.381613 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.381625 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.382828 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.382832 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.384637 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.384652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.385851 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.385855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.385981 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.386004 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:31:33.386073 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 17:31:33.386167 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":24}
I20260923 17:31:33.386538 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.380494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:40891 (local address 127.8.29.194:44349)
0923 17:31:33.380679 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.380683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.380802 (+ 119us) server_negotiation.cc:407] Connection header received
0923 17:31:33.381019 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.381024 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.381104 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.381196 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:33.381757 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.382675 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.384792 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.385714 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.385856 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.385938 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.386116 (+ 178us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.386245 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:31:33.387333 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.381062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:57065 (local address 127.8.29.193:39175)
0923 17:31:33.381291 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.381297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.381703 (+ 406us) server_negotiation.cc:407] Connection header received
0923 17:31:33.381854 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.381859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.381930 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.382027 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:33.382479 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.383230 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.385290 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.386408 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.386610 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.386704 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.387063 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.387167 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:31:33.387487 23207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.380895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39175 (local address 127.8.29.195:57065)
0923 17:31:33.381604 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:33.381621 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:33.381741 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:33.382027 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:33.382030 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:33.382047 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:33.382325 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.382333 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.383383 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.383388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.385143 (+ 1755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:33.385153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.386804 (+ 1651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.386809 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:33.386919 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.386943 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:31:33.387021 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 17:31:33.387143 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"mutex_wait_us":1,"thread_start_us":345,"threads_started":1}
I20260923 17:31:33.387554 23006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39be83b677f44e209d20cb1033a01595" candidate_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7142acd7f7424aac4861cb0d52145a" is_pre_election: true
I20260923 17:31:33.388082 23006 raft_consensus.cc:2454] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee9c4d3dbbe4c2584ed16f408c4b20c in term 0.
I20260923 17:31:33.388501 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39be83b677f44e209d20cb1033a01595" candidate_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6679092b2487475e85075d068cbf7ab2" is_pre_election: true
I20260923 17:31:33.388760 23070 leader_election.cc:302] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b7142acd7f7424aac4861cb0d52145a, cee9c4d3dbbe4c2584ed16f408c4b20c; no voters:
I20260923 17:31:33.388967 22874 raft_consensus.cc:2454] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee9c4d3dbbe4c2584ed16f408c4b20c in term 0.
I20260923 17:31:33.389127 23206 raft_consensus.cc:2781] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:31:33.389274 23206 raft_consensus.cc:482] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:31:33.389389 23206 raft_consensus.cc:3037] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:33.391871 23206 raft_consensus.cc:504] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } }
I20260923 17:31:33.392513 23206 leader_election.cc:288] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [CANDIDATE]: Term 1 election: Requested vote from peers 3b7142acd7f7424aac4861cb0d52145a (127.8.29.194:44349), 6679092b2487475e85075d068cbf7ab2 (127.8.29.193:39175)
I20260923 17:31:33.392910 23006 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39be83b677f44e209d20cb1033a01595" candidate_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b7142acd7f7424aac4861cb0d52145a"
I20260923 17:31:33.393005 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "39be83b677f44e209d20cb1033a01595" candidate_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6679092b2487475e85075d068cbf7ab2"
I20260923 17:31:33.393124 23006 raft_consensus.cc:3037] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:33.393198 22874 raft_consensus.cc:3037] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:33.396262 22874 raft_consensus.cc:2454] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee9c4d3dbbe4c2584ed16f408c4b20c in term 1.
I20260923 17:31:33.396262 23006 raft_consensus.cc:2454] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee9c4d3dbbe4c2584ed16f408c4b20c in term 1.
I20260923 17:31:33.396680 23070 leader_election.cc:302] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b7142acd7f7424aac4861cb0d52145a, cee9c4d3dbbe4c2584ed16f408c4b20c; no voters:
I20260923 17:31:33.396984 23206 raft_consensus.cc:2781] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:31:33.397401 23206 raft_consensus.cc:686] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 LEADER]: Becoming Leader. State: Replica: cee9c4d3dbbe4c2584ed16f408c4b20c, State: Running, Role: LEADER
I20260923 17:31:33.398074 23206 consensus_queue.cc:237] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } }
W20260923 17:31:33.400409 22920 tablet.cc:2369] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:31:33.402971 22731 catalog_manager.cc:5495] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c reported cstate change: term changed from 0 to 1, leader changed from <none> to cee9c4d3dbbe4c2584ed16f408c4b20c (127.8.29.195). New cstate: current_term: 1 leader_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40489 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 } health_report { overall_health: UNKNOWN } } }
I20260923 17:31:33.461153 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.455975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.8.29.195:40489)
0923 17:31:33.456150 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.456155 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.456173 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:31:33.456302 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.456306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.456385 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.456505 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:31:33.456791 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.457530 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.459080 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.459495 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.459601 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.460043 (+ 442us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:31:33.460556 (+ 513us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:31:33.460660 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.460870 (+ 210us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.460947 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:31:33.469878 23006 raft_consensus.cc:1261] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [term 1 FOLLOWER]: Refusing update from remote peer cee9c4d3dbbe4c2584ed16f408c4b20c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:33.469877 22874 raft_consensus.cc:1261] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [term 1 FOLLOWER]: Refusing update from remote peer cee9c4d3dbbe4c2584ed16f408c4b20c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:33.470948 23208 consensus_queue.cc:1035] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b7142acd7f7424aac4861cb0d52145a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 44349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:33.471315 23206 consensus_queue.cc:1035] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6679092b2487475e85075d068cbf7ab2" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:33.482438 23210 mvcc.cc:205] Tried to move back new op lower bound from 7332596504441413632 to 7332596504162463744. Current Snapshot: MvccSnapshot[applied={T|T < 7332596504441413632}]
I20260923 17:31:33.488909 23211 mvcc.cc:205] Tried to move back new op lower bound from 7332596504441413632 to 7332596504162463744. Current Snapshot: MvccSnapshot[applied={T|T < 7332596504441413632 or (T in {7332596504441413632})}]
I20260923 17:31:33.495965 22732 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:31:33.496344 22732 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46180:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:31:33.500743 22732 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Sending DeleteTablet for 3 replicas of tablet 39be83b677f44e209d20cb1033a01595
I20260923 17:31:33.501991 23117 tablet_service.cc:1514] Processing DeleteTablet for tablet 39be83b677f44e209d20cb1033a01595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:33 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:48938
I20260923 17:31:33.502080 22854 tablet_service.cc:1514] Processing DeleteTablet for tablet 39be83b677f44e209d20cb1033a01595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:33 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:40774
I20260923 17:31:33.502218 22986 tablet_service.cc:1514] Processing DeleteTablet for tablet 39be83b677f44e209d20cb1033a01595 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:33 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:38330
I20260923 17:31:33.503001 23221 tablet_replica.cc:330] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: stopping tablet replica
I20260923 17:31:33.503202 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 22770
I20260923 17:31:33.503611 23221 raft_consensus.cc:2229] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:31:33.504338 23221 raft_consensus.cc:2258] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:33.511428 23221 ts_tablet_manager.cc:1905] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:33.514521 23220 tablet_replica.cc:330] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: stopping tablet replica
I20260923 17:31:33.515177 23220 raft_consensus.cc:2229] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:31:33.515952 23220 raft_consensus.cc:2258] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:33.518356 23220 ts_tablet_manager.cc:1905] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:33.522399 23222 tablet_replica.cc:330] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: stopping tablet replica
I20260923 17:31:33.523170 23221 ts_tablet_manager.cc:1918] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:33.523255 23222 raft_consensus.cc:2229] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:31:33.523414 23221 log.cc:1196] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/39be83b677f44e209d20cb1033a01595
I20260923 17:31:33.523849 23222 raft_consensus.cc:2258] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:33.524111 23221 ts_tablet_manager.cc:1939] T 39be83b677f44e209d20cb1033a01595 P 6679092b2487475e85075d068cbf7ab2: Deleting consensus metadata
I20260923 17:31:33.526163 23220 ts_tablet_manager.cc:1918] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:33.526388 23220 log.cc:1196] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/39be83b677f44e209d20cb1033a01595
I20260923 17:31:33.526400 22717 catalog_manager.cc:4841] TS 6679092b2487475e85075d068cbf7ab2 (127.8.29.193:39175): tablet 39be83b677f44e209d20cb1033a01595 (table test-table [id=183a82f9f51a45da95f1eb7ce894a55e]) successfully deleted
I20260923 17:31:33.527040 23222 ts_tablet_manager.cc:1905] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:33.527086 23220 ts_tablet_manager.cc:1939] T 39be83b677f44e209d20cb1033a01595 P cee9c4d3dbbe4c2584ed16f408c4b20c: Deleting consensus metadata
I20260923 17:31:33.528534 22720 catalog_manager.cc:4841] TS cee9c4d3dbbe4c2584ed16f408c4b20c (127.8.29.195:40489): tablet 39be83b677f44e209d20cb1033a01595 (table test-table [id=183a82f9f51a45da95f1eb7ce894a55e]) successfully deleted
I20260923 17:31:33.529685 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.504840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.8.29.193:39175)
0923 17:31:33.505068 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.505076 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.505101 (+ 25us) server_negotiation.cc:407] Connection header received
0923 17:31:33.505175 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.505182 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.505285 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.505408 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:33.507576 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.508983 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.511023 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.511839 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.512004 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.521985 (+ 9981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:33.522012 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:33.522035 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:33.522091 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:33.526116 (+ 4025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.527161 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.527172 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.527176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.527382 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.527855 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.527861 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.527864 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.528355 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:33.528662 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.529023 (+ 361us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.529268 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:31:33.530409 22914 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:33.534228 23222 ts_tablet_manager.cc:1918] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:33.534418 23222 log.cc:1196] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/39be83b677f44e209d20cb1033a01595
I20260923 17:31:33.535122 23222 ts_tablet_manager.cc:1939] T 39be83b677f44e209d20cb1033a01595 P 3b7142acd7f7424aac4861cb0d52145a: Deleting consensus metadata
I20260923 17:31:33.537168 22717 catalog_manager.cc:4841] TS 3b7142acd7f7424aac4861cb0d52145a (127.8.29.194:44349): tablet 39be83b677f44e209d20cb1033a01595 (table test-table [id=183a82f9f51a45da95f1eb7ce894a55e]) successfully deleted
I20260923 17:31:33.623256 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 22770
I20260923 17:31:33.644479 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 22923
I20260923 17:31:33.660245 23195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.645238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.8.29.194:44349)
0923 17:31:33.645480 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.645487 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.645509 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:31:33.645573 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.645578 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.645669 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.645785 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:33.647230 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.648417 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.649919 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.650593 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.650723 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.653365 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:33.653393 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:33.653415 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:33.653459 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:33.657189 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.658016 (+ 827us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.658026 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.658030 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.658182 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.658640 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.658646 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.658649 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.659116 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:33.659405 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.659683 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.659906 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260923 17:31:33.661006 23046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:33.756976 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 22923
I20260923 17:31:33.777985 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 23053
I20260923 17:31:33.793572 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:33.778892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48960 (local address 127.8.29.195:40489)
0923 17:31:33.779095 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:33.779102 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:33.779152 (+ 50us) server_negotiation.cc:407] Connection header received
0923 17:31:33.779222 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:33.779227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:33.779310 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:33.779414 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:33.780715 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.781918 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.783294 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:33.783970 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:33.784109 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:33.786618 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:33.786640 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:33.786660 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:33.786706 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:33.790513 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.791254 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.791265 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.791268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.791485 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:33.791820 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:33.791826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:33.791833 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:33.792302 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:33.792640 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:33.792887 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 17:31:33.793116 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:31:33.794401 23177 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:33.914394 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 23053
I20260923 17:31:33.930054 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 22700
I20260923 17:31:33.931311 22763 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:34.045091 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 22700
I20260923 17:31:34.063413 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:32825
--webserver_interface=127.8.29.254
--webserver_port=46249
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:32825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:34.392397 23227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:34.392733 23227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:34.392850 23227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:34.400312 23227 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:34.400444 23227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:34.400516 23227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:34.400595 23227 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:34.400650 23227 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:34.410925 23227 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:32825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=46249
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:34.412261 23227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:34.413556 23227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:34.424721 23233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:34.424758 23235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:34.424796 23232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:34.425601 23227 server_base.cc:1034] running on GCE node
I20260923 17:31:34.426400 23227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:34.427846 23227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:34.429054 23227 hybrid_clock.cc:648] HybridClock initialized: now 1790184694428992 us; error 54 us; skew 500 ppm
I20260923 17:31:34.429382 23227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:34 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184694, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:34.432387 23227 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:31:34.433820 23227 webserver.cc:460] Webserver started at http://127.8.29.254:46249/ using document root <none> and password file <none>
I20260923 17:31:34.434587 23227 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:34.434710 23227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:34.440469 23227 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:31:34.443605 23242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:34.445256 23227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:31:34.445453 23227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "2efa957b67ab41ecb8822193aa2053df"
format_stamp: "Formatted at 2026-09-23 17:31:28 on dist-test-slave-p7jt"
server_key: "b14cf4a70ec6711b503d8391ee4130a6"
server_key_iv: "226d5857b1fcba3fed365cebb92fbdee"
server_key_version: "encryptionkey@0"
I20260923 17:31:34.446527 23227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:34.467381 23227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:34.471760 23227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:34.472033 23227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:34.490227 23227 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:32825
I20260923 17:31:34.490233 23293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:32825 every 8 connection(s)
I20260923 17:31:34.491822 23227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:31:34.496245 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 23227
I20260923 17:31:34.497201 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:39175
--local_ip_for_outbound_sockets=127.8.29.193
--tserver_master_addrs=127.8.29.254:32825
--webserver_port=43429
--webserver_interface=127.8.29.193
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:31:34.499270 23294 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:31:34.502668 23294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Bootstrap starting.
I20260923 17:31:34.531018 23294 log.cc:824] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:34.552802 23294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:31:34.553556 23294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df: Bootstrap complete.
I20260923 17:31:34.567627 23294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:34.568899 23294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2efa957b67ab41ecb8822193aa2053df, State: Initialized, Role: FOLLOWER
I20260923 17:31:34.569674 23294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:34.569959 23294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:31:34.570083 23294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:31:34.570253 23294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:31:34.575796 23294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:34.576413 23294 leader_election.cc:302] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2efa957b67ab41ecb8822193aa2053df; no voters:
I20260923 17:31:34.577099 23294 leader_election.cc:288] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:31:34.577243 23298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:31:34.578316 23298 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [term 3 LEADER]: Becoming Leader. State: Replica: 2efa957b67ab41ecb8822193aa2053df, State: Running, Role: LEADER
I20260923 17:31:34.578855 23298 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } }
I20260923 17:31:34.579172 23294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:31:34.581091 23300 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2efa957b67ab41ecb8822193aa2053df. Latest consensus state: current_term: 3 leader_uuid: "2efa957b67ab41ecb8822193aa2053df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } } }
I20260923 17:31:34.581461 23300 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:34.581952 23299 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2efa957b67ab41ecb8822193aa2053df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2efa957b67ab41ecb8822193aa2053df" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 32825 } } }
I20260923 17:31:34.582192 23299 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2efa957b67ab41ecb8822193aa2053df [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:34.585207 23304 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:31:34.604368 23304 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:31:34.606439 23304 catalog_manager.cc:1259] Loaded cluster ID: 8599245691b64b1c8c5541508ae482a5
I20260923 17:31:34.606667 23304 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 17:31:34.609212 23304 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f62531889ad google::LogMessage::Fail() at ??:0
@ 0x7f625318b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f62531884de google::LogMessage::Flush() at ??:0
@ 0x7f6253189a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f6268bbb512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f6268bb8976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f6268c40610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f6268c40439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6255dd4345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6255dd791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f6255dd7799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f6255db2eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f625908c6db start_thread at ??:0
@ 0x7f625202261f clone at ??:0
W20260923 17:31:34.901840 23296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:34.902143 23296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:34.902266 23296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:34.909951 23296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:34.910079 23296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:34.910224 23296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:31:34.920362 23296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:39175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=43429
--tserver_master_addrs=127.8.29.254:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:34.921756 23296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:34.923127 23296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:34.936548 23338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:34.936576 23336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:34.936506 23335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:34.937335 23296 server_base.cc:1034] running on GCE node
I20260923 17:31:34.937986 23296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:34.939038 23296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:34.940263 23296 hybrid_clock.cc:648] HybridClock initialized: now 1790184694940209 us; error 47 us; skew 500 ppm
I20260923 17:31:34.940604 23296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:34 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184694, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:34.943810 23296 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:31:34.945372 23296 webserver.cc:460] Webserver started at http://127.8.29.193:43429/ using document root <none> and password file <none>
I20260923 17:31:34.946043 23296 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:34.946165 23296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:34.951957 23296 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260923 17:31:34.954959 23345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:34.956480 23296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:31:34.956655 23296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "6679092b2487475e85075d068cbf7ab2"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "9c8dc2e2e5a562a14e9c677a16fa7b65"
server_key_iv: "855b92e3e70909d2a51d555fda3ab12d"
server_key_version: "encryptionkey@0"
I20260923 17:31:34.957690 23296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:34.997537 23296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:35.002359 23296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:35.002898 23296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:35.004051 23296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:35.005522 23296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:35.005626 23296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:35.005739 23296 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:35.005816 23296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:35.038293 23296 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:39175
I20260923 17:31:35.038411 23457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:39175 every 8 connection(s)
I20260923 17:31:35.039852 23296 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:31:35.044994 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:35.043039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:49879)
0923 17:31:35.043835 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:35.044062 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":327,"threads_started":1}
W20260923 17:31:35.045886 23458 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:32825 (0 consecutive failures): Network error: Failed to ping master at 127.8.29.254:32825: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
I20260923 17:31:35.046066 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 23296
I20260923 17:31:35.046828 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:44349
--local_ip_for_outbound_sockets=127.8.29.194
--tserver_master_addrs=127.8.29.254:32825
--webserver_port=39189
--webserver_interface=127.8.29.194
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:31:35.046864 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:35.046514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:48673)
0923 17:31:35.046669 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:35.046744 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 17:31:35.047829 23460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:35.047484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:55109)
0923 17:31:35.047625 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:35.047702 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:31:35.048080 23458 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:31:35.374827 23461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:35.375218 23461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:35.375331 23461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:35.382628 23461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:35.382743 23461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:35.382886 23461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:31:35.392802 23461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:44349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=39189
--tserver_master_addrs=127.8.29.254:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:35.394064 23461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:35.395416 23461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:35.408442 23467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:35.408504 23466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:35.408504 23469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:35.409152 23461 server_base.cc:1034] running on GCE node
I20260923 17:31:35.409780 23461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:35.410784 23461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:35.412017 23461 hybrid_clock.cc:648] HybridClock initialized: now 1790184695411959 us; error 44 us; skew 500 ppm
I20260923 17:31:35.412370 23461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:35 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184695, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:35.415664 23461 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:31:35.417201 23461 webserver.cc:460] Webserver started at http://127.8.29.194:39189/ using document root <none> and password file <none>
I20260923 17:31:35.417932 23461 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:35.418051 23461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:35.423796 23461 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:31:35.426759 23476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:35.428179 23461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:31:35.428344 23461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "3b7142acd7f7424aac4861cb0d52145a"
format_stamp: "Formatted at 2026-09-23 17:31:29 on dist-test-slave-p7jt"
server_key: "0e0c5445eb4e967c3732d69ca0ec608e"
server_key_iv: "c63ab582bba415480d23369be6dd0d33"
server_key_version: "encryptionkey@0"
I20260923 17:31:35.429333 23461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:35.441361 23461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:35.445652 23461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:35.445936 23461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:35.446988 23461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:35.448305 23461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:35.448398 23461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:35.448498 23461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:35.448565 23461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:35.478699 23461 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:44349
I20260923 17:31:35.478778 23588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:44349 every 8 connection(s)
I20260923 17:31:35.480304 23461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:31:35.485292 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:35.483451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:49091)
0923 17:31:35.484187 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:35.484392 (+ 205us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":300,"threads_started":1}
W20260923 17:31:35.485997 23589 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:32825 (0 consecutive failures): Network error: Failed to ping master at 127.8.29.254:32825: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
I20260923 17:31:35.486819 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:35.486492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:55927)
0923 17:31:35.486638 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:35.486703 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 17:31:35.487608 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:35.487289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:46597)
0923 17:31:35.487429 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:35.487502 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260923 17:31:35.487840 23589 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:31:35.489794 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 23461
I20260923 17:31:35.490396 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:40489
--local_ip_for_outbound_sockets=127.8.29.195
--tserver_master_addrs=127.8.29.254:32825
--webserver_port=37743
--webserver_interface=127.8.29.195
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:35.811735 23592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:35.812038 23592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:35.812177 23592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:35.819481 23592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:35.819615 23592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:35.819765 23592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:31:35.829491 23592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:40489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=37743
--tserver_master_addrs=127.8.29.254:32825
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:35.830875 23592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:35.832221 23592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:35.845036 23597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:35.845068 23600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:35.845005 23598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:35.845779 23592 server_base.cc:1034] running on GCE node
I20260923 17:31:35.846367 23592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:35.847352 23592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:35.848562 23592 hybrid_clock.cc:648] HybridClock initialized: now 1790184695848509 us; error 45 us; skew 500 ppm
I20260923 17:31:35.848862 23592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:35 dist-test-slave-p7jt krb5kdc[22150](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184695, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:35.851888 23592 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:31:35.853304 23592 webserver.cc:460] Webserver started at http://127.8.29.195:37743/ using document root <none> and password file <none>
I20260923 17:31:35.853971 23592 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:35.854081 23592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:35.859323 23592 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:31:35.861970 23607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:35.863358 23592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:31:35.863531 23592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "cee9c4d3dbbe4c2584ed16f408c4b20c"
format_stamp: "Formatted at 2026-09-23 17:31:30 on dist-test-slave-p7jt"
server_key: "1eedaf935529bea2cc320f438b799f93"
server_key_iv: "1006c8aec00fbc00d06003b427baeaf3"
server_key_version: "encryptionkey@0"
I20260923 17:31:35.864497 23592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:35.885836 23592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:35.890043 23592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:35.890313 23592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:35.891376 23592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:35.892769 23592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:35.892877 23592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:35.892992 23592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:35.893061 23592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:35.924623 23592 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:40489
I20260923 17:31:35.924679 23719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:40489 every 8 connection(s)
I20260923 17:31:35.926070 23592 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:31:35.931294 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:35.929317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:33719)
0923 17:31:35.930064 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:35.930318 (+ 254us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":294,"threads_started":1}
I20260923 17:31:35.932034 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 23592
W20260923 17:31:35.932037 23720 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:32825 (0 consecutive failures): Network error: Failed to ping master at 127.8.29.254:32825: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
I20260923 17:31:35.933182 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:35.932713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:36375)
0923 17:31:35.932942 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:35.933030 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260923 17:31:35.934135 23722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:35.933799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:45167)
0923 17:31:35.933941 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:35.934022 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 17:31:35.934361 23720 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:31:36.049597 23724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:36.048710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:48999)
0923 17:31:36.049235 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:36.049360 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":330,"threads_started":1}
I20260923 17:31:36.489176 23725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:36.488375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:42235)
0923 17:31:36.488856 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:36.488942 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":236,"threads_started":1}
I20260923 17:31:36.935832 23726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:36.934933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:56791)
0923 17:31:36.935446 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:36.935541 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":262,"threads_started":1}
I20260923 17:31:37.051342 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:37.050563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:56761)
0923 17:31:37.051059 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:37.051153 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":249,"threads_started":1}
I20260923 17:31:37.490869 23728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:37.490109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:37227)
0923 17:31:37.490608 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:37.490693 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":234,"threads_started":1}
I20260923 17:31:37.937700 23729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:37.936895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:43637)
0923 17:31:37.937427 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:37.937520 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":250,"threads_started":1}
I20260923 17:31:38.052983 23730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:38.052231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:40939)
0923 17:31:38.052719 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:38.052802 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":266,"threads_started":1}
I20260923 17:31:38.492631 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:38.491756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:57991)
0923 17:31:38.492306 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:38.492418 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":283,"threads_started":1}
I20260923 17:31:38.939561 23732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:38.938811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:50895)
0923 17:31:38.939289 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:38.939377 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":227,"threads_started":1}
I20260923 17:31:39.054919 23733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:39.054060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:47123)
0923 17:31:39.054643 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:39.054736 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":277,"threads_started":1}
I20260923 17:31:39.494567 23734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:39.493635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:56275)
0923 17:31:39.494167 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:39.494283 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":253,"threads_started":1}
I20260923 17:31:39.941483 23735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:39.940662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:55081)
0923 17:31:39.941172 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:39.941269 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":245,"threads_started":1}
I20260923 17:31:40.056646 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:40.055910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:53273)
0923 17:31:40.056382 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:40.056467 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":238,"threads_started":1}
I20260923 17:31:40.496467 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:40.495757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:33843)
0923 17:31:40.496211 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:40.496295 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":233,"threads_started":1}
I20260923 17:31:40.943313 23738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:40.942535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:48613)
0923 17:31:40.943040 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:40.943132 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":254,"threads_started":1}
I20260923 17:31:41.058629 23739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:41.057706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:44229)
0923 17:31:41.058279 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:41.058391 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":277,"threads_started":1}
I20260923 17:31:41.498131 23740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:41.497357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:33739)
0923 17:31:41.497832 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:41.497957 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":241,"threads_started":1}
I20260923 17:31:41.945003 23741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:41.944203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:35653)
0923 17:31:41.944719 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:41.944818 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":274,"threads_started":1}
I20260923 17:31:42.060492 23742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:42.059678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:34581)
0923 17:31:42.060176 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:42.060277 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":212,"threads_started":1}
I20260923 17:31:42.499928 23743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:42.499095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:40465)
0923 17:31:42.499607 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:42.499702 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":215,"threads_started":1}
I20260923 17:31:42.946844 23744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:42.945959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:52161)
0923 17:31:42.946514 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:42.946611 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":231,"threads_started":1}
I20260923 17:31:43.062312 23745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:43.061524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:46221)
0923 17:31:43.061989 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:43.062081 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":223,"threads_started":1}
I20260923 17:31:43.501674 23746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:43.500900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:35597)
0923 17:31:43.501406 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:43.501503 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":232,"threads_started":1}
I20260923 17:31:43.948560 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:43.947831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:38061)
0923 17:31:43.948300 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:43.948396 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":230,"threads_started":1}
I20260923 17:31:44.063982 23748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:44.063183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:34663)
0923 17:31:44.063699 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:44.063798 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":256,"threads_started":1}
I20260923 17:31:44.503496 23749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:44.502713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:52687)
0923 17:31:44.503198 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:44.503290 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":225,"threads_started":1}
I20260923 17:31:44.950217 23750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:44.949471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:59513)
0923 17:31:44.949955 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:44.950039 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":238,"threads_started":1}
I20260923 17:31:45.065591 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:45.064898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:56193)
0923 17:31:45.065338 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:45.065428 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260923 17:31:45.505134 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:45.504399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:44269)
0923 17:31:45.504872 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:45.504965 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":253,"threads_started":1}
I20260923 17:31:45.952212 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:45.951408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:58991)
0923 17:31:45.951928 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:45.952020 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":289,"threads_started":1}
I20260923 17:31:46.067397 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:46.066533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:36445)
0923 17:31:46.067048 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:46.067169 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":258,"threads_started":1}
I20260923 17:31:46.506985 23755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:46.506122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:35599)
0923 17:31:46.506673 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:46.506797 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":272,"threads_started":1}
I20260923 17:31:46.954104 23756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:46.953267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:59547)
0923 17:31:46.953818 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:46.953927 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":287,"threads_started":1}
I20260923 17:31:47.069067 23757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:47.068340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:43903)
0923 17:31:47.068786 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:47.068890 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":226,"threads_started":1}
I20260923 17:31:47.508853 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:47.508067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:43469)
0923 17:31:47.508608 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:47.508691 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":312,"threads_started":1}
I20260923 17:31:47.955874 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:47.955119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:55295)
0923 17:31:47.955625 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:47.955715 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":241,"threads_started":1}
I20260923 17:31:48.070973 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:48.070089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:53015)
0923 17:31:48.070656 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:48.070768 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":265,"threads_started":1}
I20260923 17:31:48.510620 23761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:48.509773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:41601)
0923 17:31:48.510271 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:48.510406 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":244,"threads_started":1}
I20260923 17:31:48.957625 23762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:48.956805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:32975)
0923 17:31:48.957345 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:48.957441 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":276,"threads_started":1}
I20260923 17:31:49.072595 23763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:49.071882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:59559)
0923 17:31:49.072311 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:49.072395 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":215,"threads_started":1}
I20260923 17:31:49.512247 23764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:49.511483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:33731)
0923 17:31:49.511972 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:49.512063 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":375,"thread_start_us":229,"threads_started":1}
I20260923 17:31:49.959358 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:49.958603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:60559)
0923 17:31:49.959076 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:49.959165 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260923 17:31:50.074721 23766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:50.073806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.193:49535)
0923 17:31:50.074369 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:50.074508 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":275,"threads_started":1}
I20260923 17:31:50.514209 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:50.513453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.194:50851)
0923 17:31:50.513923 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:50.514033 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":259,"threads_started":1}
I20260923 17:31:50.935595 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 23296
I20260923 17:31:50.958071 23768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:50.936859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33392 (local address 127.8.29.193:39175)
0923 17:31:50.937737 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:50.937758 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:50.937921 (+ 163us) server_negotiation.cc:407] Connection header received
0923 17:31:50.939701 (+ 1780us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:50.939724 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:50.940181 (+ 457us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:50.940552 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:50.942069 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:50.943370 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:50.944618 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:50.945310 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:50.945484 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:50.951874 (+ 6390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:50.951883 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:50.951901 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:50.951958 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:50.954825 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:50.955514 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:50.955522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:50.955529 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:50.955691 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:50.956069 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:50.956073 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:50.956075 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:50.956609 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:50.956919 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:50.957374 (+ 455us) server_negotiation.cc:299] Negotiation successful
0923 17:31:50.957795 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":537,"threads_started":1}
I20260923 17:31:50.959444 23453 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:50.961405 23770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:50.960464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:32825 (local address 127.8.29.195:44397)
0923 17:31:50.960995 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:50.961150 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":245,"threads_started":1}
I20260923 17:31:51.044220 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 23296
I20260923 17:31:51.057979 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 23461
I20260923 17:31:51.075904 23771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:51.058885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58420 (local address 127.8.29.194:44349)
0923 17:31:51.059424 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:51.059438 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:51.059474 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:31:51.061008 (+ 1534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:51.061017 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:51.061455 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:51.061796 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:51.063431 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:51.064672 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:51.065952 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:51.066777 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:51.066962 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:51.069596 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:51.069604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:51.069622 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:51.069666 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:51.072588 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:51.073262 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:51.073270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:51.073278 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:51.073451 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:51.073813 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:51.073818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:51.073820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:51.074435 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:51.074796 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:51.075192 (+ 396us) server_negotiation.cc:299] Negotiation successful
0923 17:31:51.075611 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":247,"threads_started":1}
I20260923 17:31:51.077319 23584 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:51.160670 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 23461
I20260923 17:31:51.175361 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 23592
I20260923 17:31:51.193141 23773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:51.176271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52842 (local address 127.8.29.195:40489)
0923 17:31:51.176895 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:51.176908 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:51.176950 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:31:51.178605 (+ 1655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:51.178617 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:51.179061 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:51.179409 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:51.180799 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:51.182067 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:51.183411 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:51.184108 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:51.184285 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:51.186906 (+ 2621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:51.186915 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:51.186931 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:51.186971 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:51.189893 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:51.190565 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:51.190572 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:51.190580 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:51.190750 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:51.191127 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:51.191131 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:51.191133 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:51.191678 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:51.191976 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:51.192414 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 17:31:51.192847 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":287,"threads_started":1}
I20260923 17:31:51.194619 23715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:51.278028 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 23592
2026-09-23T17:31:51Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24391 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:31:51 dist-test-slave-p7jt krb5kdc[23777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:31:51 dist-test-slave-p7jt krb5kdc[23777](info): set up 2 sockets
Sep 23 17:31:51 dist-test-slave-p7jt krb5kdc[23777](info): commencing operation
krb5kdc: starting...
W20260923 17:31:52.298976 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:31:52 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:31:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:31:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:31:52.537732 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39585
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:52.881107 23790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:52.881449 23790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:52.881584 23790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:52.889107 23790 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:31:52.889226 23790 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:31:52.889289 23790 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:31:52.889371 23790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:31:52.889446 23790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:52.889539 23790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:52.889595 23790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:31:52.889668 23790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:31:52.899796 23790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:52.901108 23790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:52.902478 23790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:52.915201 23795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:52.915128 23798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:52.915175 23796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:52.915848 23790 server_base.cc:1034] running on GCE node
I20260923 17:31:52.916673 23790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:52.918151 23790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:52.919377 23790 hybrid_clock.cc:648] HybridClock initialized: now 1790184712919277 us; error 77 us; skew 500 ppm
I20260923 17:31:52.919723 23790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:52 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184712, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:52.922837 23790 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:31:52.924314 23790 webserver.cc:460] Webserver started at http://127.8.29.254:33225/ using document root <none> and password file <none>
I20260923 17:31:52.925031 23790 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:52.925120 23790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:52.925395 23790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:52.928517 23790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "f24ef07f9fdf473db34e2b0f8adfbb21"
format_stamp: "Formatted at 2026-09-23 17:31:52 on dist-test-slave-p7jt"
server_key: "59813de4e1af49f97106899d9a5c6754"
server_key_iv: "902abc7443730b55bdaf0f6ae4b7c2b0"
server_key_version: "encryptionkey@0"
I20260923 17:31:52.929283 23790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "f24ef07f9fdf473db34e2b0f8adfbb21"
format_stamp: "Formatted at 2026-09-23 17:31:52 on dist-test-slave-p7jt"
server_key: "59813de4e1af49f97106899d9a5c6754"
server_key_iv: "902abc7443730b55bdaf0f6ae4b7c2b0"
server_key_version: "encryptionkey@0"
I20260923 17:31:52.934859 23790 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:31:52.938740 23805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:52.940308 23790 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:31:52.940466 23790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "f24ef07f9fdf473db34e2b0f8adfbb21"
format_stamp: "Formatted at 2026-09-23 17:31:52 on dist-test-slave-p7jt"
server_key: "59813de4e1af49f97106899d9a5c6754"
server_key_iv: "902abc7443730b55bdaf0f6ae4b7c2b0"
server_key_version: "encryptionkey@0"
I20260923 17:31:52.940671 23790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:52.959784 23790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:52.964403 23790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:52.964658 23790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:52.983681 23790 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:39585
I20260923 17:31:52.983703 23856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:39585 every 8 connection(s)
I20260923 17:31:52.985067 23790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:31:52.988981 23857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:52.991036 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 23790
I20260923 17:31:52.991279 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:31:52.992074 8311 external_mini_cluster.cc:1428] Setting key 73ab17cecb8563d35b2ca3b7b0764d7e
I20260923 17:31:53.001580 23857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Bootstrap starting.
I20260923 17:31:53.005134 23857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:53.006214 23857 log.cc:824] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:31:53 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:53.009444 23857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: No bootstrap required, opened a new log
I20260923 17:31:53.015686 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:52.994437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41124 (local address 127.8.29.254:39585)
0923 17:31:52.995319 (+ 882us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:52.995336 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:52.995380 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:31:52.997147 (+ 1767us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:52.997173 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:52.997618 (+ 445us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:52.998038 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:52.999761 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:53.001104 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:53.002564 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:53.003296 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:53.003489 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:53.007944 (+ 4455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:53.007955 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:53.007974 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:53.008016 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:53.011111 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:53.012072 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:53.012083 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:53.012090 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:53.012274 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:53.012688 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:53.012691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:53.012693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:53.013328 (+ 635us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:53.013564 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:53.014058 (+ 494us) server_negotiation.cc:299] Negotiation successful
0923 17:31:53.014550 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":352,"threads_started":1}
I20260923 17:31:53.018363 23857 raft_consensus.cc:348] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:31:53.018818 23857 raft_consensus.cc:374] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:53.018921 23857 raft_consensus.cc:729] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24ef07f9fdf473db34e2b0f8adfbb21, State: Initialized, Role: FOLLOWER
I20260923 17:31:53.019481 23857 consensus_queue.cc:260] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:31:53.019724 23857 raft_consensus.cc:388] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:31:53.019857 23857 raft_consensus.cc:482] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:31:53.020011 23857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:53.022390 23857 raft_consensus.cc:504] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:31:53.022955 23857 leader_election.cc:302] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f24ef07f9fdf473db34e2b0f8adfbb21; no voters:
I20260923 17:31:53.023554 23857 leader_election.cc:288] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:31:53.023680 23862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:31:53.024374 23862 raft_consensus.cc:686] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 1 LEADER]: Becoming Leader. State: Replica: f24ef07f9fdf473db34e2b0f8adfbb21, State: Running, Role: LEADER
I20260923 17:31:53.024912 23862 consensus_queue.cc:237] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:31:53.025558 23857 sys_catalog.cc:564] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:31:53.028115 23864 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f24ef07f9fdf473db34e2b0f8adfbb21. Latest consensus state: current_term: 1 leader_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } } }
I20260923 17:31:53.028488 23864 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:53.028918 23863 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } } }
I20260923 17:31:53.029183 23863 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: This master's current role is: LEADER
I20260923 17:31:53.040129 23870 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:31:53.045331 23870 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:31:53.054513 23870 catalog_manager.cc:1347] Generated new cluster ID: 0096a76561194380b5baf65e50be7352
I20260923 17:31:53.054648 23870 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:31:53.062368 23870 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:31:53.064378 23870 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:31:53.077672 23870 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Generated new TSK 0
I20260923 17:31:53.078665 23870 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:31:53.142809 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39585
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:53.487835 23883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:53.488195 23883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:53.488344 23883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:53.495997 23883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:53.496157 23883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:53.496289 23883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:31:53.506603 23883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:53.507936 23883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:53.509346 23883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:53.523263 23891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:53.523315 23888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:53.523294 23889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:53.524061 23883 server_base.cc:1034] running on GCE node
I20260923 17:31:53.524751 23883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:53.525842 23883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:53.527055 23883 hybrid_clock.cc:648] HybridClock initialized: now 1790184713526990 us; error 60 us; skew 500 ppm
I20260923 17:31:53.527410 23883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:53 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:53.530860 23883 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:31:53.532644 23883 webserver.cc:460] Webserver started at http://127.8.29.193:33027/ using document root <none> and password file <none>
I20260923 17:31:53.533418 23883 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:53.533553 23883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:53.533864 23883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:53.537243 23883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "ce71beefcc5e4c27a37c0200869b2d10"
format_stamp: "Formatted at 2026-09-23 17:31:53 on dist-test-slave-p7jt"
server_key: "6634c5cbb14d09699e8361808dfbcd93"
server_key_iv: "c8ee2ad072154fa2549c7ff8871ac1f9"
server_key_version: "encryptionkey@0"
I20260923 17:31:53.538076 23883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "ce71beefcc5e4c27a37c0200869b2d10"
format_stamp: "Formatted at 2026-09-23 17:31:53 on dist-test-slave-p7jt"
server_key: "6634c5cbb14d09699e8361808dfbcd93"
server_key_iv: "c8ee2ad072154fa2549c7ff8871ac1f9"
server_key_version: "encryptionkey@0"
I20260923 17:31:53.544008 23883 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260923 17:31:53.548106 23898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:53.549736 23883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:31:53.549934 23883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "ce71beefcc5e4c27a37c0200869b2d10"
format_stamp: "Formatted at 2026-09-23 17:31:53 on dist-test-slave-p7jt"
server_key: "6634c5cbb14d09699e8361808dfbcd93"
server_key_iv: "c8ee2ad072154fa2549c7ff8871ac1f9"
server_key_version: "encryptionkey@0"
I20260923 17:31:53.550155 23883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:53.571069 23883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:53.576092 23883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:53.576442 23883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:53.577593 23883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:53.579121 23883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:53.579214 23883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:53.579355 23883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:53.579416 23883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:53.613454 23883 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:37087
I20260923 17:31:53.613500 24010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:37087 every 8 connection(s)
I20260923 17:31:53.614930 23883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:31:53.621095 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 23883
I20260923 17:31:53.621295 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:31:53.621920 8311 external_mini_cluster.cc:1428] Setting key 4c1eefe19b672343b4a94baaa7d1e7b9
Sep 23 17:31:53 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184713, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:53.636613 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:53.617653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:59873 (local address 127.8.29.254:39585)
0923 17:31:53.618107 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:53.618114 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:53.619279 (+ 1165us) server_negotiation.cc:407] Connection header received
0923 17:31:53.621178 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:53.621184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:53.621291 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:53.621397 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:53.623293 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:53.624178 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:53.625910 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:53.626376 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:53.626526 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:53.630416 (+ 3890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:53.630421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:53.630425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:53.630495 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:53.633007 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:53.633837 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:53.633847 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:53.633850 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:53.634048 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:53.634632 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:53.634636 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:53.634638 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:53.634970 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:53.635120 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:53.635972 (+ 852us) server_negotiation.cc:299] Negotiation successful
0923 17:31:53.636225 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":210,"threads_started":1}
I20260923 17:31:53.637637 24014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:53.618285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:59873)
0923 17:31:53.619100 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:53.619141 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:53.620864 (+ 1723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:53.621721 (+ 857us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:53.621731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:53.622343 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:53.623106 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:53.623127 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:53.624351 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:53.624356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:53.625762 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:53.625777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:53.626654 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:53.626662 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:53.626915 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:53.627596 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:53.627626 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:53.630113 (+ 2487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:53.633176 (+ 3063us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:53.633188 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:53.633204 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:53.633670 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:53.634182 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:53.634187 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:53.634190 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:53.634498 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:53.635129 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:53.635138 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:53.635536 (+ 398us) client_negotiation.cc:769] Sending connection context
0923 17:31:53.635836 (+ 300us) client_negotiation.cc:240] Negotiation successful
0923 17:31:53.636431 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":348,"threads_started":1}
I20260923 17:31:53.639353 24011 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39585
I20260923 17:31:53.639746 24011 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:53.640763 24011 heartbeater.cc:511] Master 127.8.29.254:39585 requested a full tablet report, sending...
I20260923 17:31:53.643590 23822 ts_manager.cc:194] Registered new tserver with Master: ce71beefcc5e4c27a37c0200869b2d10 (127.8.29.193:37087)
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:31:53.645351 23822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:59873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:31:53.673926 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39585
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:53.993932 24017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:53.994211 24017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:53.994335 24017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:54.001806 24017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:54.001941 24017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:54.002089 24017 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:31:54.012161 24017 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:54.013504 24017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:54.014879 24017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:54.027933 24025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:54.028007 24023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:54.028023 24022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:54.029186 24017 server_base.cc:1034] running on GCE node
I20260923 17:31:54.029798 24017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:54.030771 24017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:54.031977 24017 hybrid_clock.cc:648] HybridClock initialized: now 1790184714031927 us; error 38 us; skew 500 ppm
I20260923 17:31:54.032287 24017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:54 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:54.035476 24017 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:31:54.036859 24017 webserver.cc:460] Webserver started at http://127.8.29.194:46675/ using document root <none> and password file <none>
I20260923 17:31:54.037554 24017 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:54.037669 24017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:54.037966 24017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:54.041021 24017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "d8616d2dab584f489b21e6bd3003031a"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "f1d88d35c75fab8661b0ad17e61f761d"
server_key_iv: "9ba47311c65295510243d63a07e04feb"
server_key_version: "encryptionkey@0"
I20260923 17:31:54.041757 24017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "d8616d2dab584f489b21e6bd3003031a"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "f1d88d35c75fab8661b0ad17e61f761d"
server_key_iv: "9ba47311c65295510243d63a07e04feb"
server_key_version: "encryptionkey@0"
I20260923 17:31:54.047140 24017 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260923 17:31:54.050503 24032 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:54.051858 24017 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:31:54.052028 24017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "d8616d2dab584f489b21e6bd3003031a"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "f1d88d35c75fab8661b0ad17e61f761d"
server_key_iv: "9ba47311c65295510243d63a07e04feb"
server_key_version: "encryptionkey@0"
I20260923 17:31:54.052217 24017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:54.076205 24017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:54.080514 24017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:54.080783 24017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:54.081753 24017 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:54.083112 24017 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:54.083189 24017 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:54.083304 24017 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:54.083356 24017 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:54.113018 24017 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:43395
I20260923 17:31:54.113098 24144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:43395 every 8 connection(s)
I20260923 17:31:54.114259 24017 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:31:54.116178 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 24017
I20260923 17:31:54.116364 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:31:54.116849 8311 external_mini_cluster.cc:1428] Setting key dbf2a71fed7581ac4b9a873dcc355c37
Sep 23 17:31:54 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:54.137099 24013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.116980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:48767 (local address 127.8.29.254:39585)
0923 17:31:54.117189 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.117196 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.118539 (+ 1343us) server_negotiation.cc:407] Connection header received
0923 17:31:54.121024 (+ 2485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.121028 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.121100 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.121189 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:54.123234 (+ 2045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.123996 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.126236 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.126766 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.126866 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.130625 (+ 3759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:54.130632 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:54.130637 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:54.130679 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:54.133664 (+ 2985us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:54.134581 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:54.134587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:54.134590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:54.134782 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:54.135221 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:54.135226 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:54.135230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:54.135614 (+ 384us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:54.135752 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:54.136522 (+ 770us) server_negotiation.cc:299] Negotiation successful
0923 17:31:54.136755 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:31:54.137967 24147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.117469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:48767)
0923 17:31:54.118372 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:54.118418 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:54.120676 (+ 2258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:54.121464 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:54.121476 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:54.122202 (+ 726us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:54.123048 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.123074 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.124164 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.124169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.126101 (+ 1932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.126116 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.127000 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.127006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.127278 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.127990 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:54.128020 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:54.130313 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:54.133842 (+ 3529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:54.133850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:54.133865 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:54.134404 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:54.134907 (+ 503us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:54.134910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:54.134913 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:54.135107 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:54.135759 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:54.135766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:54.136157 (+ 391us) client_negotiation.cc:769] Sending connection context
0923 17:31:54.136398 (+ 241us) client_negotiation.cc:240] Negotiation successful
0923 17:31:54.136812 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":394,"threads_started":1}
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
I20260923 17:31:54.148342 24145 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39585
I20260923 17:31:54.148716 24145 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:54.149502 24145 heartbeater.cc:511] Master 127.8.29.254:39585 requested a full tablet report, sending...
I20260923 17:31:54.151168 23822 ts_manager.cc:194] Registered new tserver with Master: d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395)
I20260923 17:31:54.152014 23822 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:48767
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:31:54.177580 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39585
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:31:54.532210 24150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:31:54.532511 24150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:31:54.532640 24150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:31:54.540120 24150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:31:54.540231 24150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:31:54.540369 24150 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:31:54.550681 24150 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:31:54.552074 24150 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:31:54.553332 24150 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:31:54.566655 24155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:54.566705 24156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:31:54.566617 24158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:31:54.567525 24150 server_base.cc:1034] running on GCE node
I20260923 17:31:54.568152 24150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:31:54.569156 24150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:31:54.570358 24150 hybrid_clock.cc:648] HybridClock initialized: now 1790184714570302 us; error 52 us; skew 500 ppm
I20260923 17:31:54.570689 24150 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:31:54 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:31:54.573850 24150 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:31:54.575461 24150 webserver.cc:460] Webserver started at http://127.8.29.195:43749/ using document root <none> and password file <none>
I20260923 17:31:54.576153 24150 fs_manager.cc:359] Metadata directory not provided
I20260923 17:31:54.576254 24150 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:31:54.576532 24150 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:31:54.579466 24150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "5774c239fb4d4924be99efd11c8c84fc"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "2f6e90d04760851324e4d73bc68f8871"
server_key_iv: "d6c9246ce26deca692b930e14f6b0366"
server_key_version: "encryptionkey@0"
I20260923 17:31:54.580183 24150 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "5774c239fb4d4924be99efd11c8c84fc"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "2f6e90d04760851324e4d73bc68f8871"
server_key_iv: "d6c9246ce26deca692b930e14f6b0366"
server_key_version: "encryptionkey@0"
I20260923 17:31:54.585368 24150 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:31:54.588876 24165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:54.590327 24150 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:31:54.590555 24150 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "5774c239fb4d4924be99efd11c8c84fc"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "2f6e90d04760851324e4d73bc68f8871"
server_key_iv: "d6c9246ce26deca692b930e14f6b0366"
server_key_version: "encryptionkey@0"
I20260923 17:31:54.590785 24150 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:31:54.606328 24150 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:31:54.611090 24150 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:31:54.611402 24150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:31:54.612437 24150 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:31:54.614132 24150 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:31:54.614220 24150 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:54.614352 24150 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:31:54.614423 24150 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:31:54.644748 24150 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:45417
I20260923 17:31:54.644819 24277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:45417 every 8 connection(s)
I20260923 17:31:54.646080 24150 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:31:54.649022 24011 heartbeater.cc:503] Master 127.8.29.254:39585 was elected leader, sending a full tablet report...
I20260923 17:31:54.653970 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 24150
I20260923 17:31:54.654170 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:31:54.654639 8311 external_mini_cluster.cc:1428] Setting key 0544bafa6d4aaf390ecefd11eca5a25b
Sep 23 17:31:54 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184714, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:31:54.666973 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.648633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:58475 (local address 127.8.29.254:39585)
0923 17:31:54.649091 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.649098 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.650206 (+ 1108us) server_negotiation.cc:407] Connection header received
0923 17:31:54.652255 (+ 2049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.652261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.652343 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.652435 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:54.654201 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.655209 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.657126 (+ 1917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.657626 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.657744 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.661073 (+ 3329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:54.661078 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:54.661081 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:54.661114 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:54.663677 (+ 2563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:54.664475 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:54.664481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:54.664486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:54.664627 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:54.665189 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:54.665193 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:54.665195 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:54.665480 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:54.665624 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:54.666343 (+ 719us) server_negotiation.cc:299] Negotiation successful
0923 17:31:54.666601 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":212,"threads_started":1}
I20260923 17:31:54.667956 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.649208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:58475)
0923 17:31:54.650056 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:54.650101 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:54.651989 (+ 1888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:54.652701 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:54.652710 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:54.653240 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:31:54.653995 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.654022 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.655358 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.655362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.656994 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.657009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.657855 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.657861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.658074 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.658704 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:31:54.658728 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:31:54.660848 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:31:54.663859 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:54.663867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:54.663882 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:54.664347 (+ 465us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:54.664837 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:31:54.664846 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:31:54.664849 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:31:54.665057 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:31:54.665618 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:31:54.665626 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:31:54.665978 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 17:31:54.666223 (+ 245us) client_negotiation.cc:240] Negotiation successful
0923 17:31:54.666810 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":376,"threads_started":1}
I20260923 17:31:54.669382 24278 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39585
I20260923 17:31:54.669689 24278 heartbeater.cc:464] Registering TS with master...
I20260923 17:31:54.670431 24278 heartbeater.cc:511] Master 127.8.29.254:39585 requested a full tablet report, sending...
I20260923 17:31:54.671881 23821 ts_manager.cc:194] Registered new tserver with Master: 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
I20260923 17:31:54.672756 23821 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:58475
I20260923 17:31:54.682330 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:31:54.699164 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.686860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41130 (local address 127.8.29.254:39585)
0923 17:31:54.687034 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.687039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.687467 (+ 428us) server_negotiation.cc:407] Connection header received
0923 17:31:54.687616 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.687619 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.687685 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.687772 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:54.689218 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.689929 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.691328 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.691736 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.691832 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.694303 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:54.694308 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:54.694312 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:54.694349 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:54.696833 (+ 2484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:54.697471 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:54.697474 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:54.697477 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:54.697602 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:54.697892 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:54.697895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:54.697897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:54.698225 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:54.698405 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:54.698745 (+ 340us) server_negotiation.cc:299] Negotiation successful
0923 17:31:54.698921 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:31:54.703380 23821 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41130:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:31:54.710362 23821 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:31:54.733959 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.723778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.8.29.194:43395)
0923 17:31:54.725898 (+ 2120us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.725907 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.725935 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:31:54.726041 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.726049 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.726338 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.726564 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:54.727357 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.729054 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.731258 (+ 2204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.732699 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.732843 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.733074 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:54.733432 (+ 358us) server_negotiation.cc:299] Negotiation successful
0923 17:31:54.733615 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1943,"thread_start_us":1825,"threads_started":1}
I20260923 17:31:54.733971 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.723236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:43395 (local address 127.0.0.1:36270)
0923 17:31:54.723959 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:54.724014 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:54.724207 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:54.726610 (+ 2403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:54.726617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:54.726650 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:54.727112 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.727127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.729188 (+ 2061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.729194 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.731102 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.731113 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.732939 (+ 1826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.732945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.733163 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.733229 (+ 66us) client_negotiation.cc:769] Sending connection context
0923 17:31:54.733375 (+ 146us) client_negotiation.cc:240] Negotiation successful
0923 17:31:54.733592 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":719,"threads_started":1}
I20260923 17:31:54.735702 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.723528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:37087 (local address 127.0.0.1:39080)
0923 17:31:54.727220 (+ 3692us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:54.727238 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:54.727349 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:54.728348 (+ 999us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:54.728353 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:54.728376 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:54.728853 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.728871 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.730699 (+ 1828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.730704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.732777 (+ 2073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.732793 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.734941 (+ 2148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.734947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.735103 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.735133 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:31:54.735228 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:31:54.735357 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3564,"thread_start_us":5278,"threads_started":1}
I20260923 17:31:54.736284 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.723751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.8.29.193:37087)
0923 17:31:54.724836 (+ 1085us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.724845 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.727486 (+ 2641us) server_negotiation.cc:407] Connection header received
0923 17:31:54.727605 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.727612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.727880 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.728091 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:54.729080 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.730522 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.732975 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.734431 (+ 1456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.734578 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.734811 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:54.735307 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 17:31:54.735488 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":783,"threads_started":1}
I20260923 17:31:54.737304 24292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.725254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:45417 (local address 127.0.0.1:42714)
0923 17:31:54.729099 (+ 3845us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:54.729116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:54.729228 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:54.729961 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:54.729966 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:54.729993 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:54.730407 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.730419 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.732699 (+ 2280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.732704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.734694 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.734709 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.736664 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.736669 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.736822 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.736854 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:31:54.736927 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 17:31:54.737066 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3715,"thread_start_us":2017,"threads_started":1}
I20260923 17:31:54.738121 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.725897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.8.29.195:45417)
0923 17:31:54.726681 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.726690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.729365 (+ 2675us) server_negotiation.cc:407] Connection header received
0923 17:31:54.729483 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.729491 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.729729 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.729950 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:54.730641 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.732532 (+ 1891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.734852 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.736181 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.736305 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.736527 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:54.737625 (+ 1098us) server_negotiation.cc:299] Negotiation successful
0923 17:31:54.737775 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":416,"threads_started":1}
I20260923 17:31:54.738555 24080 tablet_service.cc:1467] Processing CreateTablet for tablet 5defbe90185841718548feb5755228fa (DEFAULT_TABLE table=test-table [id=8cd4538370f741f0a70f0f091fb731d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:54.739931 24080 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5defbe90185841718548feb5755228fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:54.740409 23946 tablet_service.cc:1467] Processing CreateTablet for tablet 5defbe90185841718548feb5755228fa (DEFAULT_TABLE table=test-table [id=8cd4538370f741f0a70f0f091fb731d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:54.742302 23946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5defbe90185841718548feb5755228fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:54.741905 24213 tablet_service.cc:1467] Processing CreateTablet for tablet 5defbe90185841718548feb5755228fa (DEFAULT_TABLE table=test-table [id=8cd4538370f741f0a70f0f091fb731d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:31:54.743391 24213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5defbe90185841718548feb5755228fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:31:54.752254 24297 tablet_bootstrap.cc:492] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: Bootstrap starting.
I20260923 17:31:54.756625 24297 tablet_bootstrap.cc:654] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:54.757731 24298 tablet_bootstrap.cc:492] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: Bootstrap starting.
I20260923 17:31:54.757994 24297 log.cc:824] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:54.761832 24298 tablet_bootstrap.cc:654] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:54.762207 24299 tablet_bootstrap.cc:492] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: Bootstrap starting.
I20260923 17:31:54.763259 24298 log.cc:824] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:54.763978 24297 tablet_bootstrap.cc:492] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: No bootstrap required, opened a new log
I20260923 17:31:54.764252 24297 ts_tablet_manager.cc:1397] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260923 17:31:54.766204 24299 tablet_bootstrap.cc:654] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: Neither blocks nor log segments found. Creating new log.
I20260923 17:31:54.767482 24299 log.cc:824] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: Log is configured to *not* fsync() on all Append() calls
I20260923 17:31:54.769202 24298 tablet_bootstrap.cc:492] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: No bootstrap required, opened a new log
I20260923 17:31:54.769501 24298 ts_tablet_manager.cc:1397] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260923 17:31:54.770641 24299 tablet_bootstrap.cc:492] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: No bootstrap required, opened a new log
I20260923 17:31:54.770907 24299 ts_tablet_manager.cc:1397] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260923 17:31:54.772380 24297 raft_consensus.cc:348] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:31:54.772783 24297 raft_consensus.cc:374] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:54.772919 24297 raft_consensus.cc:729] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8616d2dab584f489b21e6bd3003031a, State: Initialized, Role: FOLLOWER
I20260923 17:31:54.773531 24297 consensus_queue.cc:260] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:31:54.774643 24145 heartbeater.cc:503] Master 127.8.29.254:39585 was elected leader, sending a full tablet report...
I20260923 17:31:54.778918 24297 ts_tablet_manager.cc:1428] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: Time spent starting tablet: real 0.015s user 0.012s sys 0.000s
I20260923 17:31:54.780370 24298 raft_consensus.cc:348] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:31:54.780844 24298 raft_consensus.cc:374] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:54.780958 24298 raft_consensus.cc:729] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce71beefcc5e4c27a37c0200869b2d10, State: Initialized, Role: FOLLOWER
I20260923 17:31:54.781445 24299 raft_consensus.cc:348] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:31:54.781623 24298 consensus_queue.cc:260] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:31:54.781903 24299 raft_consensus.cc:374] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:31:54.782003 24299 raft_consensus.cc:729] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5774c239fb4d4924be99efd11c8c84fc, State: Initialized, Role: FOLLOWER
I20260923 17:31:54.782655 24299 consensus_queue.cc:260] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:31:54.783514 24298 ts_tablet_manager.cc:1428] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 17:31:54.783744 24278 heartbeater.cc:503] Master 127.8.29.254:39585 was elected leader, sending a full tablet report...
I20260923 17:31:54.784303 24299 ts_tablet_manager.cc:1428] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: Time spent starting tablet: real 0.013s user 0.008s sys 0.004s
I20260923 17:31:54.864183 24304 raft_consensus.cc:482] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:31:54.864508 24304 raft_consensus.cc:504] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
W20260923 17:31:54.866218 24146 tablet.cc:2369] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:31:54.866690 24304 leader_election.cc:288] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395), 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
W20260923 17:31:54.867528 24012 tablet.cc:2369] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:31:54.875932 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.867259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:51749 (local address 127.8.29.194:43395)
0923 17:31:54.867480 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.867488 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.868219 (+ 731us) server_negotiation.cc:407] Connection header received
0923 17:31:54.868496 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.868502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.868585 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.868657 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:54.869477 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.870615 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.873414 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.874752 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.874940 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.875017 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:54.875514 (+ 497us) server_negotiation.cc:299] Negotiation successful
0923 17:31:54.875660 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260923 17:31:54.875936 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.867797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:59151 (local address 127.8.29.195:45417)
0923 17:31:54.868157 (+ 360us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.868162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.868841 (+ 679us) server_negotiation.cc:407] Connection header received
0923 17:31:54.868976 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.868980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.869042 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.869121 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:31:54.869715 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.870793 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.873429 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.874767 (+ 1338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.874943 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.875018 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:54.875514 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 17:31:54.875659 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226}
I20260923 17:31:54.876174 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.867092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:43395 (local address 127.8.29.193:51749)
0923 17:31:54.868096 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:54.868116 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:54.868290 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:54.868863 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:54.868868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:54.868890 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:54.869230 (+ 340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.869243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.870750 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.870754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.873182 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.873205 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.875032 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.875039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.875286 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.875340 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 17:31:54.875453 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 17:31:54.875706 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":154,"thread_start_us":659,"threads_started":1}
I20260923 17:31:54.876194 24307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.867240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:45417 (local address 127.8.29.193:59151)
0923 17:31:54.868712 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0923 17:31:54.868739 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:31:54.868863 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:31:54.869180 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:31:54.869185 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:31:54.869214 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:31:54.869523 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.869531 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.870911 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.870920 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.873183 (+ 2263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:31:54.873205 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.875044 (+ 1839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.875049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:31:54.875279 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.875337 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 17:31:54.875442 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 17:31:54.875705 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1341,"mutex_wait_us":90,"thread_start_us":1131,"threads_started":1}
I20260923 17:31:54.877506 24233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5defbe90185841718548feb5755228fa" candidate_uuid: "ce71beefcc5e4c27a37c0200869b2d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5774c239fb4d4924be99efd11c8c84fc" is_pre_election: true
I20260923 17:31:54.877516 24100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5defbe90185841718548feb5755228fa" candidate_uuid: "ce71beefcc5e4c27a37c0200869b2d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8616d2dab584f489b21e6bd3003031a" is_pre_election: true
I20260923 17:31:54.878108 24233 raft_consensus.cc:2454] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce71beefcc5e4c27a37c0200869b2d10 in term 0.
I20260923 17:31:54.878108 24100 raft_consensus.cc:2454] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce71beefcc5e4c27a37c0200869b2d10 in term 0.
I20260923 17:31:54.878921 23902 leader_election.cc:302] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce71beefcc5e4c27a37c0200869b2d10, d8616d2dab584f489b21e6bd3003031a; no voters:
I20260923 17:31:54.879383 24304 raft_consensus.cc:2781] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:31:54.879535 24304 raft_consensus.cc:482] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:31:54.879624 24304 raft_consensus.cc:3037] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:54.881729 24304 raft_consensus.cc:504] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:31:54.882313 24304 leader_election.cc:288] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 1 election: Requested vote from peers d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395), 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
I20260923 17:31:54.882817 24100 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5defbe90185841718548feb5755228fa" candidate_uuid: "ce71beefcc5e4c27a37c0200869b2d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8616d2dab584f489b21e6bd3003031a"
I20260923 17:31:54.882859 24233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5defbe90185841718548feb5755228fa" candidate_uuid: "ce71beefcc5e4c27a37c0200869b2d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5774c239fb4d4924be99efd11c8c84fc"
I20260923 17:31:54.882992 24100 raft_consensus.cc:3037] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:54.883065 24233 raft_consensus.cc:3037] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:31:54.885264 24100 raft_consensus.cc:2454] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce71beefcc5e4c27a37c0200869b2d10 in term 1.
I20260923 17:31:54.885381 24233 raft_consensus.cc:2454] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce71beefcc5e4c27a37c0200869b2d10 in term 1.
I20260923 17:31:54.885746 23902 leader_election.cc:302] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5774c239fb4d4924be99efd11c8c84fc, ce71beefcc5e4c27a37c0200869b2d10, d8616d2dab584f489b21e6bd3003031a; no voters:
I20260923 17:31:54.886071 24304 raft_consensus.cc:2781] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:31:54.886564 24304 raft_consensus.cc:686] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 1 LEADER]: Becoming Leader. State: Replica: ce71beefcc5e4c27a37c0200869b2d10, State: Running, Role: LEADER
I20260923 17:31:54.887032 24304 consensus_queue.cc:237] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:31:54.891151 23821 catalog_manager.cc:5495] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce71beefcc5e4c27a37c0200869b2d10 (127.8.29.193). New cstate: current_term: 1 leader_uuid: "ce71beefcc5e4c27a37c0200869b2d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } health_report { overall_health: UNKNOWN } } }
W20260923 17:31:54.897651 24279 tablet.cc:2369] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:31:54.928958 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.923441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.8.29.193:37087)
0923 17:31:54.923642 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.923648 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.923667 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:31:54.923892 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.923896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.923993 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.924124 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:31:54.924503 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.925210 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.926875 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.927309 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.927404 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:54.927852 (+ 448us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:31:54.928377 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:31:54.928491 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:54.928607 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 17:31:54.928680 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:31:54.937353 24233 raft_consensus.cc:1261] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [term 1 FOLLOWER]: Refusing update from remote peer ce71beefcc5e4c27a37c0200869b2d10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:54.937393 24100 raft_consensus.cc:1261] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [term 1 FOLLOWER]: Refusing update from remote peer ce71beefcc5e4c27a37c0200869b2d10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:31:54.938283 24308 consensus_queue.cc:1035] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:54.938629 24304 consensus_queue.cc:1035] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:31:54.949383 24311 mvcc.cc:205] Tried to move back new op lower bound from 7332596592373440512 to 7332596592180903936. Current Snapshot: MvccSnapshot[applied={T|T < 7332596592373440512}]
I20260923 17:31:54.962426 23821 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:31:54.962729 23821 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41130:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:31:54.966439 23821 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Sending DeleteTablet for 3 replicas of tablet 5defbe90185841718548feb5755228fa
I20260923 17:31:54.967749 23946 tablet_service.cc:1514] Processing DeleteTablet for tablet 5defbe90185841718548feb5755228fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:54 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:39080
I20260923 17:31:54.967742 24213 tablet_service.cc:1514] Processing DeleteTablet for tablet 5defbe90185841718548feb5755228fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:54 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:42714
I20260923 17:31:54.967782 24080 tablet_service.cc:1514] Processing DeleteTablet for tablet 5defbe90185841718548feb5755228fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:31:54 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:36270
I20260923 17:31:54.972164 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 23883
I20260923 17:31:54.980710 24321 tablet_replica.cc:330] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: stopping tablet replica
I20260923 17:31:54.981307 24321 raft_consensus.cc:2229] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:31:54.982090 24321 raft_consensus.cc:2258] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:54.983142 24322 tablet_replica.cc:330] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: stopping tablet replica
I20260923 17:31:54.983860 24322 raft_consensus.cc:2229] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:31:54.984547 24322 raft_consensus.cc:2258] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:54.987334 24322 ts_tablet_manager.cc:1905] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:54.991065 24320 tablet_replica.cc:330] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: stopping tablet replica
I20260923 17:31:54.991550 24320 raft_consensus.cc:2229] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:31:54.992141 24320 raft_consensus.cc:2258] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:31:54.994768 24321 ts_tablet_manager.cc:1905] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:55.000063 24320 ts_tablet_manager.cc:1905] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:31:55.004024 24322 ts_tablet_manager.cc:1918] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:55.004235 24322 log.cc:1196] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/5defbe90185841718548feb5755228fa
I20260923 17:31:55.004791 24322 ts_tablet_manager.cc:1939] T 5defbe90185841718548feb5755228fa P d8616d2dab584f489b21e6bd3003031a: Deleting consensus metadata
I20260923 17:31:55.007387 24320 ts_tablet_manager.cc:1918] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:55.007586 24320 log.cc:1196] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/5defbe90185841718548feb5755228fa
I20260923 17:31:55.008121 24320 ts_tablet_manager.cc:1939] T 5defbe90185841718548feb5755228fa P 5774c239fb4d4924be99efd11c8c84fc: Deleting consensus metadata
I20260923 17:31:55.010275 23807 catalog_manager.cc:4841] TS 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417): tablet 5defbe90185841718548feb5755228fa (table test-table [id=8cd4538370f741f0a70f0f091fb731d1]) successfully deleted
Sep 23 17:31:55 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:31:55.013958 24321 ts_tablet_manager.cc:1918] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:31:55.014158 24321 log.cc:1196] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/5defbe90185841718548feb5755228fa
I20260923 17:31:55.014803 24321 ts_tablet_manager.cc:1939] T 5defbe90185841718548feb5755228fa P ce71beefcc5e4c27a37c0200869b2d10: Deleting consensus metadata
I20260923 17:31:55.014930 23809 catalog_manager.cc:4841] TS d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395): tablet 5defbe90185841718548feb5755228fa (table test-table [id=8cd4538370f741f0a70f0f091fb731d1]) successfully deleted
I20260923 17:31:55.016599 23806 catalog_manager.cc:4841] TS ce71beefcc5e4c27a37c0200869b2d10 (127.8.29.193:37087): tablet 5defbe90185841718548feb5755228fa (table test-table [id=8cd4538370f741f0a70f0f091fb731d1]) successfully deleted
I20260923 17:31:55.022138 24293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:54.973510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39090 (local address 127.8.29.193:37087)
0923 17:31:54.973692 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:54.973698 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:54.991473 (+ 17775us) server_negotiation.cc:407] Connection header received
0923 17:31:54.991847 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:54.991853 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:54.991950 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:54.992024 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:54.994391 (+ 2367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.995608 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.997772 (+ 2164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:54.998530 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:54.998637 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:55.014204 (+ 15567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:55.014223 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:55.014246 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:55.014306 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:55.018932 (+ 4626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:55.019809 (+ 877us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:55.019829 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:55.019833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:55.020034 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:55.020411 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:55.020416 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:55.020420 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:55.020938 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:55.021304 (+ 366us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:55.021472 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 17:31:55.021735 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:31:55.022989 24006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:55.130438 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 23883
I20260923 17:31:55.146618 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 24017
Sep 23 17:31:55 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:31:55.161134 24294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:55.147341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36278 (local address 127.8.29.194:43395)
0923 17:31:55.147527 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:55.147532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:55.147549 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:31:55.147641 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:55.147645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:55.147720 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:55.147805 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:55.149024 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:55.149776 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:55.151172 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:55.151558 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:55.151645 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:55.155756 (+ 4111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:55.155771 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:55.155788 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:55.155825 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:55.158675 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:55.159392 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:55.159400 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:55.159402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:55.159523 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:55.159815 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:55.159819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:55.159821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:55.160203 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:55.160446 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:55.160622 (+ 176us) server_negotiation.cc:299] Negotiation successful
0923 17:31:55.160780 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:31:55.161828 24140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:55.250397 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 24017
I20260923 17:31:55.264786 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 24150
Sep 23 17:31:55 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:31:55.280000 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:31:55.265625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42716 (local address 127.8.29.195:45417)
0923 17:31:55.265815 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:31:55.265820 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:31:55.265841 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:31:55.266024 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:31:55.266029 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:31:55.266101 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:31:55.266188 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:31:55.267644 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:55.268387 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:55.269642 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:31:55.270049 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:31:55.270142 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:31:55.274358 (+ 4216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:31:55.274375 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:31:55.274392 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:31:55.274433 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:31:55.277170 (+ 2737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:55.277932 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:55.277940 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:55.277942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:55.278090 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:31:55.278548 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:31:55.278553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:31:55.278556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:31:55.278964 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:31:55.279185 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 17:31:55.279475 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 17:31:55.279653 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:31:55.280664 24273 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:55.369668 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 24150
I20260923 17:31:55.383490 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 23790
I20260923 17:31:55.384691 23852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:31:55.496218 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 23790
I20260923 17:32:15.514678 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39585
--webserver_interface=127.8.29.254
--webserver_port=33225
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:32:15.842841 24328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:15.843144 24328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:15.843233 24328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:15.850521 24328 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:32:15.850627 24328 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:32:15.850677 24328 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:32:15.850731 24328 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:32:15.850776 24328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:15.850841 24328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:15.850885 24328 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:32:15.850943 24328 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:32:15.860878 24328 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:15.862140 24328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:15.863456 24328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:15.885555 24334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:15.885496 24335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:15.885530 24337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:15.886308 24328 server_base.cc:1034] running on GCE node
I20260923 17:32:15.887269 24328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:15.888955 24328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:15.890240 24328 hybrid_clock.cc:648] HybridClock initialized: now 1790184735890160 us; error 49 us; skew 500 ppm
I20260923 17:32:15.890707 24328 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:15 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184735, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:15.894271 24328 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:32:15.896054 24328 webserver.cc:460] Webserver started at http://127.8.29.254:33225/ using document root <none> and password file <none>
I20260923 17:32:15.896875 24328 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:15.896981 24328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:15.903357 24328 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260923 17:32:15.906401 24344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:15.907934 24328 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:32:15.908087 24328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "f24ef07f9fdf473db34e2b0f8adfbb21"
format_stamp: "Formatted at 2026-09-23 17:31:52 on dist-test-slave-p7jt"
server_key: "59813de4e1af49f97106899d9a5c6754"
server_key_iv: "902abc7443730b55bdaf0f6ae4b7c2b0"
server_key_version: "encryptionkey@0"
I20260923 17:32:15.909168 24328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:15.938803 24328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:15.943344 24328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:15.943647 24328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:15.961966 24328 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:39585
I20260923 17:32:15.961987 24395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:39585 every 8 connection(s)
I20260923 17:32:15.963678 24328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:32:15.968255 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 24328
I20260923 17:32:15.969257 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:37087
--local_ip_for_outbound_sockets=127.8.29.193
--tserver_master_addrs=127.8.29.254:39585
--webserver_port=33027
--webserver_interface=127.8.29.193
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:32:15.972671 24396 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:32:15.977799 24396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Bootstrap starting.
I20260923 17:32:15.998878 24396 log.cc:824] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:16.011812 24396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:32:16.012344 24396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Bootstrap complete.
I20260923 17:32:16.021996 24396 raft_consensus.cc:348] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:32:16.022949 24396 raft_consensus.cc:729] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24ef07f9fdf473db34e2b0f8adfbb21, State: Initialized, Role: FOLLOWER
I20260923 17:32:16.023689 24396 consensus_queue.cc:260] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:32:16.023931 24396 raft_consensus.cc:388] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:32:16.024055 24396 raft_consensus.cc:482] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:32:16.024209 24396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 1 FOLLOWER]: Advancing to term 2
I20260923 17:32:16.028316 24396 raft_consensus.cc:504] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:32:16.029016 24396 leader_election.cc:302] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f24ef07f9fdf473db34e2b0f8adfbb21; no voters:
I20260923 17:32:16.029745 24396 leader_election.cc:288] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 17:32:16.029927 24400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 17:32:16.031032 24400 raft_consensus.cc:686] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 2 LEADER]: Becoming Leader. State: Replica: f24ef07f9fdf473db34e2b0f8adfbb21, State: Running, Role: LEADER
I20260923 17:32:16.031729 24400 consensus_queue.cc:237] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:32:16.032193 24396 sys_catalog.cc:564] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:32:16.034299 24402 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f24ef07f9fdf473db34e2b0f8adfbb21. Latest consensus state: current_term: 2 leader_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } } }
I20260923 17:32:16.034732 24402 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: This master's current role is: LEADER
I20260923 17:32:16.034415 24401 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } } }
I20260923 17:32:16.034909 24401 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: This master's current role is: LEADER
I20260923 17:32:16.041692 24407 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:32:16.051801 24407 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:32:16.053571 24407 catalog_manager.cc:1259] Loaded cluster ID: 0096a76561194380b5baf65e50be7352
I20260923 17:32:16.053786 24407 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:32:16.058255 24407 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:32:16.060137 24407 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Loaded TSK: 0
I20260923 17:32:16.092320 24407 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Generated new TSK 1
I20260923 17:32:16.092783 24407 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 17:32:16.326442 24398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:16.326787 24398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:16.326911 24398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:16.334609 24398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:16.334722 24398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:16.334862 24398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:32:16.345477 24398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:37087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=33027
--tserver_master_addrs=127.8.29.254:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:16.346863 24398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:16.348160 24398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:16.362697 24424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:16.362697 24423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:16.363418 24426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:16.363780 24398 server_base.cc:1034] running on GCE node
I20260923 17:32:16.364405 24398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:16.365381 24398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:16.366560 24398 hybrid_clock.cc:648] HybridClock initialized: now 1790184736366486 us; error 60 us; skew 500 ppm
I20260923 17:32:16.366896 24398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:16 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184736, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:16.369997 24398 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:32:16.371490 24398 webserver.cc:460] Webserver started at http://127.8.29.193:33027/ using document root <none> and password file <none>
I20260923 17:32:16.372150 24398 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:16.372284 24398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:16.377768 24398 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260923 17:32:16.380842 24433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:16.382339 24398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:32:16.382529 24398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "ce71beefcc5e4c27a37c0200869b2d10"
format_stamp: "Formatted at 2026-09-23 17:31:53 on dist-test-slave-p7jt"
server_key: "6634c5cbb14d09699e8361808dfbcd93"
server_key_iv: "c8ee2ad072154fa2549c7ff8871ac1f9"
server_key_version: "encryptionkey@0"
I20260923 17:32:16.383522 24398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:16.404798 24398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:16.409514 24398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:16.409852 24398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:16.410976 24398 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:32:16.412487 24398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:32:16.412673 24398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:16.412823 24398 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:32:16.412904 24398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:16.443383 24398 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:37087
I20260923 17:32:16.443459 24545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:37087 every 8 connection(s)
I20260923 17:32:16.444800 24398 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:32:16.446949 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 24398
I20260923 17:32:16.447636 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:43395
--local_ip_for_outbound_sockets=127.8.29.194
--tserver_master_addrs=127.8.29.254:39585
--webserver_port=46675
--webserver_interface=127.8.29.194
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:32:16 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184736, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:32:16.474095 24549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:16.448176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:47217)
0923 17:32:16.451028 (+ 2852us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:16.451078 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:16.452847 (+ 1769us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:16.454564 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:16.454576 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:16.455363 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:32:16.455941 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:16.455960 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:16.458264 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:16.458269 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:16.459566 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:16.459579 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:16.460977 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:16.460983 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:16.461224 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:16.461941 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:32:16.461971 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:32:16.464377 (+ 2406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:32:16.468918 (+ 4541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:16.468925 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:16.468940 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:16.469427 (+ 487us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:16.470041 (+ 614us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:16.470045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:16.470047 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:16.470241 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:16.471298 (+ 1057us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:32:16.471305 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:32:16.471629 (+ 324us) client_negotiation.cc:769] Sending connection context
0923 17:32:16.471957 (+ 328us) client_negotiation.cc:240] Negotiation successful
0923 17:32:16.472552 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2555,"thread_start_us":2335,"threads_started":1}
I20260923 17:32:16.474270 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:16.448021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:47217 (local address 127.8.29.254:39585)
0923 17:32:16.448688 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:16.448706 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:16.451228 (+ 2522us) server_negotiation.cc:407] Connection header received
0923 17:32:16.453469 (+ 2241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:16.453488 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:16.453955 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:16.454282 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:16.456165 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:16.458059 (+ 1894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:16.459704 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:16.460714 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:16.460886 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:16.464715 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:16.464727 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:16.464752 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:16.464802 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:16.468734 (+ 3932us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:16.469624 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:16.469636 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:16.469645 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:16.469910 (+ 265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:16.470367 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:16.470372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:16.470375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:16.471145 (+ 770us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:16.471378 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:16.472392 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0923 17:32:16.473073 (+ 681us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":268,"threads_started":1}
I20260923 17:32:16.477910 24546 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39585
I20260923 17:32:16.478209 24546 heartbeater.cc:464] Registering TS with master...
I20260923 17:32:16.479003 24546 heartbeater.cc:511] Master 127.8.29.254:39585 requested a full tablet report, sending...
I20260923 17:32:16.481204 24361 ts_manager.cc:194] Registered new tserver with Master: ce71beefcc5e4c27a37c0200869b2d10 (127.8.29.193:37087)
I20260923 17:32:16.482781 24361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:47217
W20260923 17:32:16.785873 24550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:16.786157 24550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:16.786293 24550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:16.793728 24550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:16.793846 24550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:16.793988 24550 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:32:16.804011 24550 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:43395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=46675
--tserver_master_addrs=127.8.29.254:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:16.805326 24550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:16.806665 24550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:16.819841 24556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:16.819779 24558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:16.820009 24555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:16.820711 24550 server_base.cc:1034] running on GCE node
I20260923 17:32:16.821368 24550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:16.822311 24550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:16.823509 24550 hybrid_clock.cc:648] HybridClock initialized: now 1790184736823440 us; error 56 us; skew 500 ppm
I20260923 17:32:16.823819 24550 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:16 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184736, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:16.826817 24550 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:32:16.828204 24550 webserver.cc:460] Webserver started at http://127.8.29.194:46675/ using document root <none> and password file <none>
I20260923 17:32:16.828884 24550 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:16.828996 24550 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:16.834689 24550 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260923 17:32:16.837483 24565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:16.838932 24550 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:32:16.839094 24550 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "d8616d2dab584f489b21e6bd3003031a"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "f1d88d35c75fab8661b0ad17e61f761d"
server_key_iv: "9ba47311c65295510243d63a07e04feb"
server_key_version: "encryptionkey@0"
I20260923 17:32:16.840090 24550 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:16.854713 24550 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:16.859066 24550 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:16.859336 24550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:16.860363 24550 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:32:16.861758 24550 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:32:16.861855 24550 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:16.861961 24550 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:32:16.862030 24550 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:16.893836 24550 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:43395
I20260923 17:32:16.893889 24677 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:43395 every 8 connection(s)
I20260923 17:32:16.895371 24550 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:32:16.902884 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 24550
I20260923 17:32:16.903646 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:45417
--local_ip_for_outbound_sockets=127.8.29.195
--tserver_master_addrs=127.8.29.254:39585
--webserver_port=43749
--webserver_interface=127.8.29.195
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 17:32:16 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184736, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:32:16.917161 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:16.898077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:46123 (local address 127.8.29.254:39585)
0923 17:32:16.898267 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:16.898272 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:16.899488 (+ 1216us) server_negotiation.cc:407] Connection header received
0923 17:32:16.901375 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:16.901381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:16.901466 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:16.901555 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:16.903579 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:16.904370 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:16.906672 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:16.907135 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:16.907246 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:16.911396 (+ 4150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:16.911402 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:16.911405 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:16.911443 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:16.913995 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:16.914853 (+ 858us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:16.914857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:16.914859 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:16.914991 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:16.915421 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:16.915425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:16.915427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:16.915722 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:16.915861 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:16.916578 (+ 717us) server_negotiation.cc:299] Negotiation successful
0923 17:32:16.916822 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:32:16.918205 24680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:16.898526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:46123)
0923 17:32:16.899293 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:16.899342 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:16.901052 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:16.901829 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:16.901848 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:16.902443 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:32:16.903350 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:16.903379 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:16.904550 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:16.904556 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:16.906500 (+ 1944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:16.906517 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:16.907387 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:16.907393 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:16.907682 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:16.908446 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:32:16.908477 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:32:16.911091 (+ 2614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:32:16.914164 (+ 3073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:16.914174 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:16.914194 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:16.914704 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:16.915099 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:16.915104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:16.915107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:16.915300 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:16.915849 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:32:16.915857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:32:16.916209 (+ 352us) client_negotiation.cc:769] Sending connection context
0923 17:32:16.916460 (+ 251us) client_negotiation.cc:240] Negotiation successful
0923 17:32:16.917041 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":305,"threads_started":1}
I20260923 17:32:16.919747 24678 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39585
I20260923 17:32:16.920076 24678 heartbeater.cc:464] Registering TS with master...
I20260923 17:32:16.920869 24678 heartbeater.cc:511] Master 127.8.29.254:39585 requested a full tablet report, sending...
I20260923 17:32:16.922435 24361 ts_manager.cc:194] Registered new tserver with Master: d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395)
I20260923 17:32:16.923640 24361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:46123
W20260923 17:32:17.242784 24681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:17.243075 24681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:17.243168 24681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:17.250438 24681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:17.250587 24681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:17.250708 24681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:32:17.260475 24681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:45417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=43749
--tserver_master_addrs=127.8.29.254:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:17.261727 24681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:17.263028 24681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:17.276072 24687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:17.276104 24686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:17.276068 24689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:17.276839 24681 server_base.cc:1034] running on GCE node
I20260923 17:32:17.277451 24681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:17.278395 24681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:17.279657 24681 hybrid_clock.cc:648] HybridClock initialized: now 1790184737279605 us; error 38 us; skew 500 ppm
I20260923 17:32:17.280000 24681 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:17 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184737, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:17.283107 24681 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:32:17.284646 24681 webserver.cc:460] Webserver started at http://127.8.29.195:43749/ using document root <none> and password file <none>
I20260923 17:32:17.285319 24681 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:17.285442 24681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:17.290830 24681 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:32:17.293516 24696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:17.294829 24681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:32:17.295001 24681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "5774c239fb4d4924be99efd11c8c84fc"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "2f6e90d04760851324e4d73bc68f8871"
server_key_iv: "d6c9246ce26deca692b930e14f6b0366"
server_key_version: "encryptionkey@0"
I20260923 17:32:17.295954 24681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:17.325187 24681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:17.329422 24681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:17.329656 24681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:17.330657 24681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:32:17.331969 24681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:32:17.332109 24681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:17.332195 24681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:32:17.332248 24681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:17.361546 24681 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:45417
I20260923 17:32:17.361599 24808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:45417 every 8 connection(s)
I20260923 17:32:17.362989 24681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:32:17.369177 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 24681
Sep 23 17:32:17 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184737, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:32:17.384713 24548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.365604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:51937 (local address 127.8.29.254:39585)
0923 17:32:17.365784 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.365788 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.366972 (+ 1184us) server_negotiation.cc:407] Connection header received
0923 17:32:17.368802 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.368810 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.368910 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.369045 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:17.370854 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.371939 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.373619 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.374165 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.374315 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.378106 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:17.378113 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:17.378119 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:17.378161 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:17.381160 (+ 2999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.382062 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.382066 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.382068 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.382185 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.382683 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.382688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.382692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.383054 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:17.383183 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.384062 (+ 879us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.384284 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65,"spinlock_wait_cycles":768}
I20260923 17:32:17.385219 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.370162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32800 (local address 127.8.29.254:39585)
0923 17:32:17.370641 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.370647 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.371088 (+ 441us) server_negotiation.cc:407] Connection header received
0923 17:32:17.371286 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.371289 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.371350 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.371427 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:17.373311 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.374142 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.375667 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.376280 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.376420 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.379471 (+ 3051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:17.379476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:17.379479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:17.379529 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:17.382358 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.383179 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.383184 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.383187 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.383312 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.383697 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.383701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.383704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.384051 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:17.384294 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.384609 (+ 315us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.384827 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"spinlock_wait_cycles":1280,"thread_start_us":222,"threads_started":1}
I20260923 17:32:17.385911 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.366060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:51937)
0923 17:32:17.366792 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.366831 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:17.368483 (+ 1652us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:17.369234 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:17.369242 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:17.369796 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:32:17.370659 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.370686 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.372100 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.372103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.373447 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.373475 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.374386 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.374391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.374641 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.375257 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:32:17.375280 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:32:17.377869 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:32:17.381304 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:17.381315 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:17.381336 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:17.381943 (+ 607us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:17.382284 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:17.382290 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:17.382294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:17.382569 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:17.383176 (+ 607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:32:17.383184 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:32:17.383596 (+ 412us) client_negotiation.cc:769] Sending connection context
0923 17:32:17.383958 (+ 362us) client_negotiation.cc:240] Negotiation successful
0923 17:32:17.384474 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":269,"threads_started":1}
I20260923 17:32:17.388118 24809 heartbeater.cc:347] Connected to a master server at 127.8.29.254:39585
I20260923 17:32:17.388405 24809 heartbeater.cc:464] Registering TS with master...
I20260923 17:32:17.389065 24809 heartbeater.cc:511] Master 127.8.29.254:39585 requested a full tablet report, sending...
I20260923 17:32:17.390340 24361 ts_manager.cc:194] Registered new tserver with Master: 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
I20260923 17:32:17.391168 24361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:51937
I20260923 17:32:17.400307 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:32:17.406306 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.401173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32812 (local address 127.8.29.254:39585)
0923 17:32:17.401355 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.401359 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.401745 (+ 386us) server_negotiation.cc:407] Connection header received
0923 17:32:17.401937 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.401941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.402076 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.402190 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:32:17.402574 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.403273 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.404640 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.405049 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.405139 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.405610 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:32:17.405809 (+ 199us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 17:32:17.406098 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:32812: token expired
Metrics: {"server-negotiator.queue_time_us":47}
W20260923 17:32:17.406418 24812 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:32812: token expired
W20260923 17:32:17.407006 24284 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.29.254:39585)
I20260923 17:32:17.419945 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.407529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.8.29.254:39585)
0923 17:32:17.407696 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.407701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.407719 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:32:17.407765 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.407770 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.407828 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.407912 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:17.409267 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.409961 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.411427 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.411805 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.411918 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.414583 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:17.414589 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:17.414592 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:17.414630 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:17.417160 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.417888 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.417892 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.417895 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.418030 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.418425 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.418429 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.418431 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.418806 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:17.419049 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.419372 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.419614 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:32:17.423821 24361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32820:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:32:17.431609 24361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:32:17.457722 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.445105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54010 (local address 127.8.29.193:37087)
0923 17:32:17.446128 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.446139 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.447375 (+ 1236us) server_negotiation.cc:407] Connection header received
0923 17:32:17.449026 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.449037 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.449352 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.449527 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:17.450619 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.452168 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.455105 (+ 2937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.456495 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.456623 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.456871 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.457226 (+ 355us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.457417 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":829,"thread_start_us":656,"threads_started":1}
I20260923 17:32:17.458575 24819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.444248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:37087 (local address 127.0.0.1:54010)
0923 17:32:17.447220 (+ 2972us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.447262 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:17.447522 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:17.449882 (+ 2360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:17.449891 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:17.449928 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:17.450373 (+ 445us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.450386 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.452379 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.452384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.454933 (+ 2549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.454952 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.456726 (+ 1774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.456732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.456952 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.457012 (+ 60us) client_negotiation.cc:769] Sending connection context
0923 17:32:17.457930 (+ 918us) client_negotiation.cc:240] Negotiation successful
0923 17:32:17.458121 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2741,"mutex_wait_us":4,"thread_start_us":2608,"threads_started":1}
I20260923 17:32:17.459133 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.444521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:43395 (local address 127.0.0.1:37036)
0923 17:32:17.451020 (+ 6499us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.451042 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:17.451278 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:17.452250 (+ 972us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:17.452255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:17.452282 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:17.452836 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.452849 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.454354 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.454359 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.456259 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.456274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.458303 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.458309 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.458503 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.458538 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:32:17.458635 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:32:17.458892 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6354,"mutex_wait_us":22,"thread_start_us":6576,"threads_started":1}
I20260923 17:32:17.459209 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.445493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.8.29.194:43395)
0923 17:32:17.446573 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.446581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.451165 (+ 4584us) server_negotiation.cc:407] Connection header received
0923 17:32:17.451498 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.451508 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.451783 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.451955 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:17.453034 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.454203 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.456449 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.457909 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.458007 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.458197 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.458690 (+ 493us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.458881 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":765,"threads_started":1}
I20260923 17:32:17.460008 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.448274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:45417 (local address 127.0.0.1:52230)
0923 17:32:17.451365 (+ 3091us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.451383 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:17.451488 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:17.453852 (+ 2364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:17.453857 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:17.453883 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:17.454268 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.454281 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.455880 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.455885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.457889 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.457903 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.459525 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.459532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.459647 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.459669 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 17:32:17.459717 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 17:32:17.459796 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2979,"thread_start_us":2462,"threads_started":1}
I20260923 17:32:17.460822 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.448518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52230 (local address 127.8.29.195:45417)
0923 17:32:17.453228 (+ 4710us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.453238 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.453271 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:32:17.453365 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.453372 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.453620 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.453841 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:17.454436 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.455731 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.458078 (+ 2347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.459278 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.460238 (+ 960us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.460442 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.460494 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.460606 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4534,"thread_start_us":4389,"threads_started":1}
I20260923 17:32:17.463210 24613 tablet_service.cc:1467] Processing CreateTablet for tablet 42bcb16875304deab15cdbb8c42dbf33 (DEFAULT_TABLE table=test-table [id=8348fc0221984b3d8bb9d9315c4987e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:32:17.463210 24481 tablet_service.cc:1467] Processing CreateTablet for tablet 42bcb16875304deab15cdbb8c42dbf33 (DEFAULT_TABLE table=test-table [id=8348fc0221984b3d8bb9d9315c4987e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:32:17.464102 24744 tablet_service.cc:1467] Processing CreateTablet for tablet 42bcb16875304deab15cdbb8c42dbf33 (DEFAULT_TABLE table=test-table [id=8348fc0221984b3d8bb9d9315c4987e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:32:17.464648 24613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bcb16875304deab15cdbb8c42dbf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:32:17.464663 24481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bcb16875304deab15cdbb8c42dbf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:32:17.465173 24744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42bcb16875304deab15cdbb8c42dbf33. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:32:17.480526 24825 tablet_bootstrap.cc:492] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: Bootstrap starting.
I20260923 17:32:17.481880 24827 tablet_bootstrap.cc:492] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: Bootstrap starting.
I20260923 17:32:17.483448 24826 tablet_bootstrap.cc:492] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Bootstrap starting.
I20260923 17:32:17.485641 24827 tablet_bootstrap.cc:654] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: Neither blocks nor log segments found. Creating new log.
I20260923 17:32:17.485641 24825 tablet_bootstrap.cc:654] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: Neither blocks nor log segments found. Creating new log.
I20260923 17:32:17.485909 24546 heartbeater.cc:503] Master 127.8.29.254:39585 was elected leader, sending a full tablet report...
I20260923 17:32:17.486553 24826 tablet_bootstrap.cc:654] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Neither blocks nor log segments found. Creating new log.
I20260923 17:32:17.487000 24827 log.cc:824] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:17.487015 24825 log.cc:824] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:17.487905 24826 log.cc:824] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:17.491204 24826 tablet_bootstrap.cc:492] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: No bootstrap required, opened a new log
I20260923 17:32:17.491459 24826 ts_tablet_manager.cc:1397] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260923 17:32:17.492713 24827 tablet_bootstrap.cc:492] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: No bootstrap required, opened a new log
I20260923 17:32:17.492996 24827 ts_tablet_manager.cc:1397] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20260923 17:32:17.493031 24825 tablet_bootstrap.cc:492] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: No bootstrap required, opened a new log
I20260923 17:32:17.493263 24825 ts_tablet_manager.cc:1397] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.002s
I20260923 17:32:17.498623 24826 raft_consensus.cc:348] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:17.499078 24826 raft_consensus.cc:374] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:32:17.499209 24826 raft_consensus.cc:729] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce71beefcc5e4c27a37c0200869b2d10, State: Initialized, Role: FOLLOWER
I20260923 17:32:17.499869 24826 consensus_queue.cc:260] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:17.501763 24826 ts_tablet_manager.cc:1428] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Time spent starting tablet: real 0.010s user 0.009s sys 0.001s
I20260923 17:32:17.503643 24827 raft_consensus.cc:348] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:17.504112 24827 raft_consensus.cc:374] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:32:17.504205 24827 raft_consensus.cc:729] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8616d2dab584f489b21e6bd3003031a, State: Initialized, Role: FOLLOWER
I20260923 17:32:17.504840 24827 consensus_queue.cc:260] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:17.504956 24825 raft_consensus.cc:348] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:17.505385 24825 raft_consensus.cc:374] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:32:17.505493 24825 raft_consensus.cc:729] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5774c239fb4d4924be99efd11c8c84fc, State: Initialized, Role: FOLLOWER
I20260923 17:32:17.505971 24678 heartbeater.cc:503] Master 127.8.29.254:39585 was elected leader, sending a full tablet report...
I20260923 17:32:17.506145 24825 consensus_queue.cc:260] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:17.506770 24827 ts_tablet_manager.cc:1428] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: Time spent starting tablet: real 0.014s user 0.011s sys 0.004s
I20260923 17:32:17.507292 24809 heartbeater.cc:503] Master 127.8.29.254:39585 was elected leader, sending a full tablet report...
I20260923 17:32:17.507928 24825 ts_tablet_manager.cc:1428] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: Time spent starting tablet: real 0.015s user 0.013s sys 0.001s
I20260923 17:32:17.510080 24831 raft_consensus.cc:482] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:32:17.510329 24831 raft_consensus.cc:504] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:17.512151 24831 leader_election.cc:288] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395), 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
I20260923 17:32:17.519400 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.512421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:49955 (local address 127.8.29.194:43395)
0923 17:32:17.512618 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.512622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.513552 (+ 930us) server_negotiation.cc:407] Connection header received
0923 17:32:17.513859 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.513866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.513935 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.513993 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:17.514914 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.515644 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.517622 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.518403 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.518610 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.518701 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.519019 (+ 318us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.519153 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:32:17.519554 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.512692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:39871 (local address 127.8.29.195:45417)
0923 17:32:17.512902 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.512907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.513758 (+ 851us) server_negotiation.cc:407] Connection header received
0923 17:32:17.514053 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.514058 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.514128 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.514213 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:17.514936 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.515647 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.517620 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.518438 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.518622 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.518701 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.519152 (+ 451us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.519313 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260923 17:32:17.519594 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.512917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:45417 (local address 127.8.29.193:39871)
0923 17:32:17.513672 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.513687 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:17.513900 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:17.514235 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:17.514239 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:17.514264 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:17.514770 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.514780 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.515777 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.515783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.517459 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.517472 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.518699 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.518704 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.518889 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.518913 (+ 24us) client_negotiation.cc:769] Sending connection context
0923 17:32:17.518990 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:32:17.519130 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":83,"thread_start_us":442,"threads_started":1}
I20260923 17:32:17.519604 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.512624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:43395 (local address 127.8.29.193:49955)
0923 17:32:17.513431 (+ 807us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.513467 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:17.513674 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:17.514206 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:17.514210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:17.514234 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:17.514768 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.514779 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.515785 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.515790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.517453 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:17.517468 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.518664 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.518670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:17.518852 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.518893 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 17:32:17.518975 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 17:32:17.519119 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"mutex_wait_us":62,"thread_start_us":389,"threads_started":1}
I20260923 17:32:17.520697 24633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bcb16875304deab15cdbb8c42dbf33" candidate_uuid: "ce71beefcc5e4c27a37c0200869b2d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8616d2dab584f489b21e6bd3003031a" is_pre_election: true
I20260923 17:32:17.520668 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bcb16875304deab15cdbb8c42dbf33" candidate_uuid: "ce71beefcc5e4c27a37c0200869b2d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5774c239fb4d4924be99efd11c8c84fc" is_pre_election: true
I20260923 17:32:17.521256 24764 raft_consensus.cc:2454] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce71beefcc5e4c27a37c0200869b2d10 in term 0.
I20260923 17:32:17.521256 24633 raft_consensus.cc:2454] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce71beefcc5e4c27a37c0200869b2d10 in term 0.
I20260923 17:32:17.521968 24435 leader_election.cc:302] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5774c239fb4d4924be99efd11c8c84fc, ce71beefcc5e4c27a37c0200869b2d10; no voters:
I20260923 17:32:17.522614 24831 raft_consensus.cc:2781] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:32:17.522768 24831 raft_consensus.cc:482] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:32:17.522868 24831 raft_consensus.cc:3037] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:32:17.525070 24831 raft_consensus.cc:504] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:17.525702 24831 leader_election.cc:288] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 1 election: Requested vote from peers d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395), 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
I20260923 17:32:17.526324 24764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bcb16875304deab15cdbb8c42dbf33" candidate_uuid: "ce71beefcc5e4c27a37c0200869b2d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5774c239fb4d4924be99efd11c8c84fc"
I20260923 17:32:17.526363 24633 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42bcb16875304deab15cdbb8c42dbf33" candidate_uuid: "ce71beefcc5e4c27a37c0200869b2d10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8616d2dab584f489b21e6bd3003031a"
I20260923 17:32:17.526587 24633 raft_consensus.cc:3037] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:32:17.526588 24764 raft_consensus.cc:3037] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:32:17.528849 24633 raft_consensus.cc:2454] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce71beefcc5e4c27a37c0200869b2d10 in term 1.
I20260923 17:32:17.529018 24764 raft_consensus.cc:2454] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce71beefcc5e4c27a37c0200869b2d10 in term 1.
I20260923 17:32:17.529516 24435 leader_election.cc:302] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5774c239fb4d4924be99efd11c8c84fc, ce71beefcc5e4c27a37c0200869b2d10; no voters:
I20260923 17:32:17.530043 24831 raft_consensus.cc:2781] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:32:17.530634 24831 raft_consensus.cc:686] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 LEADER]: Becoming Leader. State: Replica: ce71beefcc5e4c27a37c0200869b2d10, State: Running, Role: LEADER
I20260923 17:32:17.531246 24831 consensus_queue.cc:237] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:17.536280 24360 catalog_manager.cc:5495] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 reported cstate change: term changed from 0 to 1, leader changed from <none> to ce71beefcc5e4c27a37c0200869b2d10 (127.8.29.193). New cstate: current_term: 1 leader_uuid: "ce71beefcc5e4c27a37c0200869b2d10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } health_report { overall_health: UNKNOWN } } }
I20260923 17:32:17.562333 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.556950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54026 (local address 127.8.29.193:37087)
0923 17:32:17.557155 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.557160 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.557180 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:32:17.557445 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.557450 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.557538 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.557678 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:32:17.558045 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.558828 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.560183 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.560612 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.560725 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.561248 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:32:17.561758 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:32:17.561914 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.561992 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.562077 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:32:17.570909 24764 raft_consensus.cc:1261] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [term 1 FOLLOWER]: Refusing update from remote peer ce71beefcc5e4c27a37c0200869b2d10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:32:17.570906 24633 raft_consensus.cc:1261] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [term 1 FOLLOWER]: Refusing update from remote peer ce71beefcc5e4c27a37c0200869b2d10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:32:17.571885 24836 consensus_queue.cc:1035] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:32:17.572257 24831 consensus_queue.cc:1035] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:32:17.601573 24360 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:32:17.601878 24360 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32820:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:32:17.605628 24360 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Sending DeleteTablet for 3 replicas of tablet 42bcb16875304deab15cdbb8c42dbf33
I20260923 17:32:17.607005 24481 tablet_service.cc:1514] Processing DeleteTablet for tablet 42bcb16875304deab15cdbb8c42dbf33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:32:17 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:54010
I20260923 17:32:17.606993 24613 tablet_service.cc:1514] Processing DeleteTablet for tablet 42bcb16875304deab15cdbb8c42dbf33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:32:17 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:37036
I20260923 17:32:17.607584 24744 tablet_service.cc:1514] Processing DeleteTablet for tablet 42bcb16875304deab15cdbb8c42dbf33 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:32:17 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:52230
I20260923 17:32:17.607913 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 24398
I20260923 17:32:17.620114 24849 tablet_replica.cc:330] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: stopping tablet replica
I20260923 17:32:17.621346 24849 raft_consensus.cc:2229] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:32:17.622063 24849 raft_consensus.cc:2258] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:32:17.624351 24849 ts_tablet_manager.cc:1905] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:32:17.624760 24847 tablet_replica.cc:330] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: stopping tablet replica
I20260923 17:32:17.625202 24847 raft_consensus.cc:2229] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:32:17.625831 24848 tablet_replica.cc:330] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: stopping tablet replica
I20260923 17:32:17.626240 24848 raft_consensus.cc:2229] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:32:17.626938 24848 raft_consensus.cc:2258] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:32:17.627681 24847 raft_consensus.cc:2258] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:32:17.627936 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.608719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54036 (local address 127.8.29.193:37087)
0923 17:32:17.608882 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.608888 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.608961 (+ 73us) server_negotiation.cc:407] Connection header received
0923 17:32:17.609147 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.609153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.609241 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.609361 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:17.611125 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.612493 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.615968 (+ 3475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.616757 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.616885 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.620365 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:17.620384 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:17.620407 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:17.620452 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:17.624449 (+ 3997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.625426 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.625439 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.625443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.625641 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.626084 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.626090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.626093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.626655 (+ 562us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:17.626971 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.627303 (+ 332us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.627553 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:32:17.628932 24541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:17.630084 24847 ts_tablet_manager.cc:1905] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:32:17.631803 24849 ts_tablet_manager.cc:1918] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:32:17.631953 24849 log.cc:1196] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/42bcb16875304deab15cdbb8c42dbf33
I20260923 17:32:17.632395 24849 ts_tablet_manager.cc:1939] T 42bcb16875304deab15cdbb8c42dbf33 P 5774c239fb4d4924be99efd11c8c84fc: Deleting consensus metadata
I20260923 17:32:17.634055 24346 catalog_manager.cc:4841] TS 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417): tablet 42bcb16875304deab15cdbb8c42dbf33 (table test-table [id=8348fc0221984b3d8bb9d9315c4987e3]) successfully deleted
I20260923 17:32:17.637780 24847 ts_tablet_manager.cc:1918] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:32:17.637964 24847 log.cc:1196] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/42bcb16875304deab15cdbb8c42dbf33
I20260923 17:32:17.638366 24847 ts_tablet_manager.cc:1939] T 42bcb16875304deab15cdbb8c42dbf33 P d8616d2dab584f489b21e6bd3003031a: Deleting consensus metadata
I20260923 17:32:17.639971 24348 catalog_manager.cc:4841] TS d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395): tablet 42bcb16875304deab15cdbb8c42dbf33 (table test-table [id=8348fc0221984b3d8bb9d9315c4987e3]) successfully deleted
I20260923 17:32:17.730197 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 24398
I20260923 17:32:17.745154 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 24550
W20260923 17:32:17.745196 24345 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.29.193:37087 (error 108)
I20260923 17:32:17.746980 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.746591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:37087 (local address 127.0.0.1:54042)
0923 17:32:17.746751 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.746848 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 17:32:17.747186 24345 catalog_manager.cc:4584] TS ce71beefcc5e4c27a37c0200869b2d10 (127.8.29.193:37087): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 42bcb16875304deab15cdbb8c42dbf33: Network error: Client connection negotiation failed: client connection to 127.8.29.193:37087: connect: Connection refused (error 111)
I20260923 17:32:17.762975 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.746046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37044 (local address 127.8.29.194:43395)
0923 17:32:17.746216 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.746222 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.746308 (+ 86us) server_negotiation.cc:407] Connection header received
0923 17:32:17.746580 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.746585 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.746659 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.746763 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:17.748190 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.748952 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.750237 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.750691 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.750809 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.757265 (+ 6456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:17.757282 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:17.757298 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:17.757352 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:17.760136 (+ 2784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.760877 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.760886 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.760888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.761032 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.761394 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.761398 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.761400 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.761791 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:17.762068 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.762406 (+ 338us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.762658 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:32:17.763670 24673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:17.800804 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.800304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:37087 (local address 127.0.0.1:54056)
0923 17:32:17.800537 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.800634 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260923 17:32:17.801919 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.801524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:37087 (local address 127.0.0.1:54068)
0923 17:32:17.801677 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.801769 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260923 17:32:17.852588 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 24550
I20260923 17:32:17.862787 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.862108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:37087 (local address 127.0.0.1:54074)
0923 17:32:17.862412 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.862582 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260923 17:32:17.864043 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.863622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:37087 (local address 127.0.0.1:54078)
0923 17:32:17.863803 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.863897 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260923 17:32:17.866991 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 24681
I20260923 17:32:17.891983 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.867765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52242 (local address 127.8.29.195:45417)
0923 17:32:17.867921 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:17.867926 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:17.867945 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:32:17.868025 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:17.868029 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:17.868120 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:17.868201 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:17.869714 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.870532 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.871996 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:17.872478 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:17.872588 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:17.885312 (+ 12724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:17.885330 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:17.885348 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:17.885393 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:17.888788 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.889696 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.889705 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.889708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.889876 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:17.890365 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:17.890370 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:17.890372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:17.890938 (+ 566us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:17.891169 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:17.891498 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 17:32:17.891687 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:32:17.892637 24804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:17.945075 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.944589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:37087 (local address 127.0.0.1:54094)
0923 17:32:17.944805 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.944910 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260923 17:32:17.946137 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:17.945756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:37087 (local address 127.0.0.1:54102)
0923 17:32:17.945955 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:17.946029 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:37087: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260923 17:32:17.981823 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 24681
I20260923 17:32:17.995378 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 24328
I20260923 17:32:17.996501 24391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:18.110780 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 24328
I20260923 17:32:38.129657 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39585
--webserver_interface=127.8.29.254
--webserver_port=33225
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:39585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:32:38.457554 24855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:38.457857 24855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:38.457980 24855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:38.465262 24855 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 17:32:38.465379 24855 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 17:32:38.465418 24855 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 17:32:38.465487 24855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:32:38.465547 24855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:38.465638 24855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:38.465689 24855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:32:38.465744 24855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:32:38.475832 24855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:39585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=33225
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:38.477119 24855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:38.478403 24855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:38.490104 24860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:38.490123 24861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:38.490656 24863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:38.491397 24855 server_base.cc:1034] running on GCE node
I20260923 17:32:38.492242 24855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:38.493775 24855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:38.495043 24855 hybrid_clock.cc:648] HybridClock initialized: now 1790184758494960 us; error 61 us; skew 500 ppm
I20260923 17:32:38.495411 24855 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:38 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184758, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:38.503070 24855 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:32:38.504627 24855 webserver.cc:460] Webserver started at http://127.8.29.254:33225/ using document root <none> and password file <none>
I20260923 17:32:38.505326 24855 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:38.505416 24855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:38.511545 24855 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260923 17:32:38.514811 24870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:38.516331 24855 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260923 17:32:38.516494 24855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "f24ef07f9fdf473db34e2b0f8adfbb21"
format_stamp: "Formatted at 2026-09-23 17:31:52 on dist-test-slave-p7jt"
server_key: "59813de4e1af49f97106899d9a5c6754"
server_key_iv: "902abc7443730b55bdaf0f6ae4b7c2b0"
server_key_version: "encryptionkey@0"
I20260923 17:32:38.517515 24855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:38.541672 24855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:38.547653 24855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:38.547950 24855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:38.565215 24855 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:39585
I20260923 17:32:38.565249 24921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:39585 every 8 connection(s)
I20260923 17:32:38.566748 24855 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:32:38.573905 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 24855
I20260923 17:32:38.573966 24922 sys_catalog.cc:263] Verifying existing consensus state
I20260923 17:32:38.574971 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:37087
--local_ip_for_outbound_sockets=127.8.29.193
--tserver_master_addrs=127.8.29.254:39585
--webserver_port=33027
--webserver_interface=127.8.29.193
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:32:38.578572 24922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Bootstrap starting.
I20260923 17:32:38.600371 24922 log.cc:824] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:38.616999 24922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:32:38.617561 24922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Bootstrap complete.
I20260923 17:32:38.626735 24922 raft_consensus.cc:348] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:32:38.627615 24922 raft_consensus.cc:729] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f24ef07f9fdf473db34e2b0f8adfbb21, State: Initialized, Role: FOLLOWER
I20260923 17:32:38.628249 24922 consensus_queue.cc:260] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:32:38.628461 24922 raft_consensus.cc:388] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:32:38.628569 24922 raft_consensus.cc:482] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:32:38.628732 24922 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 2 FOLLOWER]: Advancing to term 3
I20260923 17:32:38.632542 24922 raft_consensus.cc:504] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:32:38.633019 24922 leader_election.cc:302] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f24ef07f9fdf473db34e2b0f8adfbb21; no voters:
I20260923 17:32:38.633651 24922 leader_election.cc:288] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 17:32:38.633783 24926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 17:32:38.634884 24926 raft_consensus.cc:686] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [term 3 LEADER]: Becoming Leader. State: Replica: f24ef07f9fdf473db34e2b0f8adfbb21, State: Running, Role: LEADER
I20260923 17:32:38.635490 24926 consensus_queue.cc:237] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } }
I20260923 17:32:38.635746 24922 sys_catalog.cc:564] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:32:38.637931 24928 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f24ef07f9fdf473db34e2b0f8adfbb21. Latest consensus state: current_term: 3 leader_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } } }
I20260923 17:32:38.638334 24928 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: This master's current role is: LEADER
I20260923 17:32:38.638959 24927 sys_catalog.cc:455] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f24ef07f9fdf473db34e2b0f8adfbb21" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39585 } } }
I20260923 17:32:38.639302 24927 sys_catalog.cc:458] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21 [sys.catalog]: This master's current role is: LEADER
I20260923 17:32:38.650437 24932 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:32:38.659765 24932 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:32:38.661492 24932 catalog_manager.cc:1259] Loaded cluster ID: 0096a76561194380b5baf65e50be7352
I20260923 17:32:38.661685 24932 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:32:38.666430 24932 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:32:38.668313 24932 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Loaded TSK: 0
I20260923 17:32:38.668573 24932 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f24ef07f9fdf473db34e2b0f8adfbb21: Loaded TSK: 1
F20260923 17:32:38.669812 24932 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f39d61599ad google::LogMessage::Fail() at ??:0
@ 0x7f39d615c88a google::LogMessage::SendToLog() at ??:0
@ 0x7f39d61594de google::LogMessage::Flush() at ??:0
I20260923 17:32:38.677711 24943 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f39d615aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f39d9ebd7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f39d9eac73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f39ebb881c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f39ebb87783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f39ebc132a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f39ebc130c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f39ebc5d5e9 std::function<>::operator()() at ??:0
@ 0x7f39ebb8bcdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f39ebb89c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f39ebc11610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f39ebc11439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f39d8da5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f39d8da891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f39d8da8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f39d8d83eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f39dc05d6db start_thread at ??:0
@ 0x7f39d4ff361f clone at ??:0
W20260923 17:32:38.949339 24924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:38.949640 24924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:38.949744 24924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:38.957270 24924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:38.957388 24924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:38.957500 24924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:32:38.968001 24924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:37087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=33027
--tserver_master_addrs=127.8.29.254:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:38.969349 24924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:38.970671 24924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:38.984480 24969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:38.984517 24970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:38.984457 24972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:38.985284 24924 server_base.cc:1034] running on GCE node
I20260923 17:32:38.985931 24924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:38.986963 24924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:38.988168 24924 hybrid_clock.cc:648] HybridClock initialized: now 1790184758988113 us; error 47 us; skew 500 ppm
I20260923 17:32:38.988493 24924 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:38 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184758, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:38.991622 24924 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:32:38.993212 24924 webserver.cc:460] Webserver started at http://127.8.29.193:33027/ using document root <none> and password file <none>
I20260923 17:32:38.993921 24924 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:38.994041 24924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:38.999957 24924 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260923 17:32:39.003054 24979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:39.004624 24924 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:32:39.004801 24924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "ce71beefcc5e4c27a37c0200869b2d10"
format_stamp: "Formatted at 2026-09-23 17:31:53 on dist-test-slave-p7jt"
server_key: "6634c5cbb14d09699e8361808dfbcd93"
server_key_iv: "c8ee2ad072154fa2549c7ff8871ac1f9"
server_key_version: "encryptionkey@0"
I20260923 17:32:39.005805 24924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:39.049917 24924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:39.054507 24924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:39.054786 24924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:39.055892 24924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:32:39.057888 24986 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260923 17:32:39.063871 24924 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 17:32:39.064028 24924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20260923 17:32:39.064172 24924 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 17:32:39.067525 24924 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 17:32:39.067621 24924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.003s sys 0.000s
I20260923 17:32:39.068008 24986 tablet_bootstrap.cc:492] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Bootstrap starting.
I20260923 17:32:39.077440 24986 log.cc:824] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:39.097792 24986 tablet_bootstrap.cc:492] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 17:32:39.098800 24986 tablet_bootstrap.cc:492] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Bootstrap complete.
I20260923 17:32:39.099432 24986 ts_tablet_manager.cc:1397] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Time spent bootstrapping tablet: real 0.032s user 0.019s sys 0.011s
I20260923 17:32:39.108909 24986 raft_consensus.cc:348] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:39.110221 24986 raft_consensus.cc:729] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce71beefcc5e4c27a37c0200869b2d10, State: Initialized, Role: FOLLOWER
I20260923 17:32:39.110491 24924 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:37087
I20260923 17:32:39.110561 25093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:37087 every 8 connection(s)
I20260923 17:32:39.111110 24986 consensus_queue.cc:260] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:39.112097 24924 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:32:39.112747 24986 ts_tablet_manager.cc:1428] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10: Time spent starting tablet: real 0.013s user 0.012s sys 0.000s
I20260923 17:32:39.115953 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 24924
I20260923 17:32:39.116643 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:43395
--local_ip_for_outbound_sockets=127.8.29.194
--tserver_master_addrs=127.8.29.254:39585
--webserver_port=46675
--webserver_interface=127.8.29.194
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:32:39.117100 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:39.115087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:34291)
0923 17:32:39.115865 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:39.116103 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":330,"threads_started":1}
W20260923 17:32:39.117844 25095 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:39585 (0 consecutive failures): Network error: Failed to ping master at 127.8.29.254:39585: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
I20260923 17:32:39.118710 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:39.118352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:41413)
0923 17:32:39.118528 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:39.118602 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 17:32:39.119580 25097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:39.119242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:49905)
0923 17:32:39.119366 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:39.119458 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 17:32:39.119796 25095 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:32:39.452678 25098 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:39.453037 25098 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:39.453290 25098 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:39.461424 25098 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:39.461585 25098 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:39.461705 25098 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:32:39.472213 25098 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:43395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=46675
--tserver_master_addrs=127.8.29.254:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:39.473578 25098 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:39.474961 25098 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:39.488986 25103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:39.488914 25106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:39.488938 25104 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:39.489425 25098 server_base.cc:1034] running on GCE node
I20260923 17:32:39.490329 25098 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:39.491389 25098 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:39.492632 25098 hybrid_clock.cc:648] HybridClock initialized: now 1790184759492570 us; error 45 us; skew 500 ppm
I20260923 17:32:39.493002 25098 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:39 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184759, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:39.496634 25098 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:32:39.498404 25098 webserver.cc:460] Webserver started at http://127.8.29.194:46675/ using document root <none> and password file <none>
I20260923 17:32:39.499212 25098 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:39.499524 25098 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:39.505533 25098 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260923 17:32:39.508798 25113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:39.510398 25098 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:32:39.510612 25098 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "d8616d2dab584f489b21e6bd3003031a"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "f1d88d35c75fab8661b0ad17e61f761d"
server_key_iv: "9ba47311c65295510243d63a07e04feb"
server_key_version: "encryptionkey@0"
I20260923 17:32:39.511721 25098 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:39.531837 25098 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:39.536387 25098 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:39.536675 25098 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:39.537753 25098 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:32:39.539289 25098 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:32:39.539402 25098 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:39.539516 25098 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:32:39.539583 25098 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:39.574400 25098 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:43395
I20260923 17:32:39.574432 25225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:43395 every 8 connection(s)
I20260923 17:32:39.575892 25098 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:32:39.580899 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 25098
I20260923 17:32:39.581278 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:39.579193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:44139)
0923 17:32:39.580031 (+ 838us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:39.580231 (+ 200us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":355,"threads_started":1}
I20260923 17:32:39.581632 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:45417
--local_ip_for_outbound_sockets=127.8.29.195
--tserver_master_addrs=127.8.29.254:39585
--webserver_port=43749
--webserver_interface=127.8.29.195
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:32:39.582115 25226 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:39585 (0 consecutive failures): Network error: Failed to ping master at 127.8.29.254:39585: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
I20260923 17:32:39.583156 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:39.582791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:46519)
0923 17:32:39.582959 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:39.583038 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 17:32:39.584156 25228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:39.583730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:39169)
0923 17:32:39.583893 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:39.583992 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 17:32:39.584898 25226 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 17:32:39.920022 25229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:39.920332 25229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:39.920428 25229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:39.928282 25229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:39.928411 25229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:39.928531 25229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:32:39.938565 25229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:45417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=43749
--tserver_master_addrs=127.8.29.254:39585
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:39.939888 25229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:39.941203 25229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:39.954439 25235 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:39.954507 25237 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:39.954403 25234 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:39.955466 25229 server_base.cc:1034] running on GCE node
I20260923 17:32:39.956168 25229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:39.957182 25229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:39.958424 25229 hybrid_clock.cc:648] HybridClock initialized: now 1790184759958353 us; error 51 us; skew 500 ppm
I20260923 17:32:39.958877 25229 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:39 dist-test-slave-p7jt krb5kdc[23777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184759, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:39.962124 25229 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:32:39.963734 25229 webserver.cc:460] Webserver started at http://127.8.29.195:43749/ using document root <none> and password file <none>
I20260923 17:32:39.964408 25229 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:39.964512 25229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:39.970187 25229 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260923 17:32:39.973186 25244 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:39.974889 25229 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:32:39.975044 25229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "5774c239fb4d4924be99efd11c8c84fc"
format_stamp: "Formatted at 2026-09-23 17:31:54 on dist-test-slave-p7jt"
server_key: "2f6e90d04760851324e4d73bc68f8871"
server_key_iv: "d6c9246ce26deca692b930e14f6b0366"
server_key_version: "encryptionkey@0"
I20260923 17:32:39.976050 25229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:39.994292 25229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:40.001058 25229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:40.001374 25229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:40.002650 25229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:32:40.004177 25229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:32:40.004271 25229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:40.004359 25229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:32:40.004436 25229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:40.036660 25229 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:45417
I20260923 17:32:40.036712 25356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:45417 every 8 connection(s)
I20260923 17:32:40.038110 25229 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:32:40.043605 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:40.041204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:33993)
0923 17:32:40.041992 (+ 788us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:40.042328 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":301,"threads_started":1}
W20260923 17:32:40.044480 25357 heartbeater.cc:644] Failed to heartbeat to 127.8.29.254:39585 (0 consecutive failures): Network error: Failed to ping master at 127.8.29.254:39585: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
I20260923 17:32:40.045550 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:40.045033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:34009)
0923 17:32:40.045328 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:40.045419 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171}
I20260923 17:32:40.046574 25359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:40.046159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:53783)
0923 17:32:40.046313 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:40.046396 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:32:40.046844 25357 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 17:32:40.048539 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 25229
I20260923 17:32:40.121182 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:40.120367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:34753)
0923 17:32:40.120835 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:40.120965 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":226,"threads_started":1}
I20260923 17:32:40.354354 25362 raft_consensus.cc:482] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:32:40.354693 25362 raft_consensus.cc:504] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:40.356462 25362 leader_election.cc:288] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395), 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
Sep 23 17:32:40 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184758, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
Sep 23 17:32:40 dist-test-slave-p7jt krb5kdc[23777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184758, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:32:40.377954 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:40.357215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:50083 (local address 127.8.29.195:45417)
0923 17:32:40.358633 (+ 1418us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:40.358648 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:40.359175 (+ 527us) server_negotiation.cc:407] Connection header received
0923 17:32:40.360694 (+ 1519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:40.360702 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:40.361041 (+ 339us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:40.361369 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:40.363069 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:40.364298 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:40.366138 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:40.367168 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:40.367354 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:40.371495 (+ 4141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:40.371502 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:40.371516 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:40.371555 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:40.374287 (+ 2732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:40.375144 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:40.375150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:40.375156 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:40.375319 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:40.375764 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:40.375769 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:40.375772 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:40.376295 (+ 523us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:40.376463 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:40.377237 (+ 774us) server_negotiation.cc:299] Negotiation successful
0923 17:32:40.377578 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1190,"thread_start_us":995,"threads_started":1}
I20260923 17:32:40.377982 25363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:40.356895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:45417 (local address 127.8.29.193:50083)
0923 17:32:40.359061 (+ 2166us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:40.359075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:40.359200 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:40.361368 (+ 2168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:40.361380 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:40.362141 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:32:40.362883 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:40.362904 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:40.364481 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:40.364485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:40.365961 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:40.365974 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:40.367666 (+ 1692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:40.367670 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:40.367801 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:40.368522 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:32:40.368540 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:32:40.371207 (+ 2667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:32:40.374436 (+ 3229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:40.374440 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:40.374442 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:40.375005 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:40.375430 (+ 425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:40.375436 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:40.375440 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:40.375652 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:40.376424 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:32:40.376431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:32:40.376747 (+ 316us) client_negotiation.cc:769] Sending connection context
0923 17:32:40.376987 (+ 240us) client_negotiation.cc:240] Negotiation successful
0923 17:32:40.377525 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2045,"mutex_wait_us":110,"spinlock_wait_cycles":166784,"thread_start_us":1894,"threads_started":1}
I20260923 17:32:40.378026 25364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:40.357002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:57881 (local address 127.8.29.194:43395)
0923 17:32:40.358941 (+ 1939us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:40.358955 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:40.358998 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:32:40.360448 (+ 1450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:40.360457 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:40.360877 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:40.361145 (+ 268us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:40.363081 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:40.364298 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:40.366103 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:40.367169 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:40.367354 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:40.371146 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:40.371158 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:40.371179 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:40.371233 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:40.374245 (+ 3012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:40.375133 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:40.375141 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:40.375147 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:40.375327 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:40.375778 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:40.375783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:40.375786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:40.376279 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:40.376464 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:40.377242 (+ 778us) server_negotiation.cc:299] Negotiation successful
0923 17:32:40.377634 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1748,"thread_start_us":1611,"threads_started":1}
I20260923 17:32:40.378878 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:40.356724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:43395 (local address 127.8.29.193:57881)
0923 17:32:40.357050 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:40.357089 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:40.357742 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:40.361369 (+ 3627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:40.361380 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:40.362140 (+ 760us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:32:40.362883 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:40.362904 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:40.364488 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:40.364491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:40.365953 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:40.365967 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:40.367412 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:40.367419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:40.367667 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:40.368373 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:32:40.368410 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:32:40.370799 (+ 2389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:32:40.374404 (+ 3605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:40.374413 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:40.374429 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:40.374986 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:40.375445 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:40.375450 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:40.375453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:40.375662 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:40.376421 (+ 759us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:32:40.376429 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:32:40.376754 (+ 325us) client_negotiation.cc:769] Sending connection context
0923 17:32:40.376986 (+ 232us) client_negotiation.cc:240] Negotiation successful
0923 17:32:40.378617 (+ 1631us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":747,"spinlock_wait_cycles":16000}
W20260923 17:32:40.381400 24981 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
W20260923 17:32:40.381696 24983 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
I20260923 17:32:40.381851 24983 leader_election.cc:302] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce71beefcc5e4c27a37c0200869b2d10; no voters: 5774c239fb4d4924be99efd11c8c84fc, d8616d2dab584f489b21e6bd3003031a
I20260923 17:32:40.382256 25362 raft_consensus.cc:2726] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:32:40.586251 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:40.585417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:52831)
0923 17:32:40.585888 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:40.586016 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":203,"threads_started":1}
I20260923 17:32:41.048069 25367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:41.047327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:42225)
0923 17:32:41.047774 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:41.047877 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":206,"threads_started":1}
I20260923 17:32:41.123005 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:41.122220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:39923)
0923 17:32:41.122741 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:41.122830 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":282,"threads_started":1}
I20260923 17:32:41.588016 25369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:41.587261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:41663)
0923 17:32:41.587722 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:41.587810 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":240,"threads_started":1}
I20260923 17:32:42.049778 25370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:42.049003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:37097)
0923 17:32:42.049499 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:42.049588 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":246,"threads_started":1}
I20260923 17:32:42.062525 25371 raft_consensus.cc:482] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:32:42.062723 25371 raft_consensus.cc:504] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:42.063366 25371 leader_election.cc:288] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395), 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
W20260923 17:32:42.064392 24983 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
W20260923 17:32:42.064545 24981 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
I20260923 17:32:42.064673 24981 leader_election.cc:302] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce71beefcc5e4c27a37c0200869b2d10; no voters: 5774c239fb4d4924be99efd11c8c84fc, d8616d2dab584f489b21e6bd3003031a
I20260923 17:32:42.064939 25371 raft_consensus.cc:2726] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:32:42.124756 25372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:42.123960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:53781)
0923 17:32:42.124452 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:42.124552 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":222,"threads_started":1}
I20260923 17:32:42.589636 25373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:42.588913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:35631)
0923 17:32:42.589386 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:42.589472 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":250,"threads_started":1}
I20260923 17:32:43.051594 25374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:43.050805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:36713)
0923 17:32:43.051266 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:43.051384 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":250,"threads_started":1}
I20260923 17:32:43.126605 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:43.125846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:57463)
0923 17:32:43.126306 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:43.126400 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":228,"threads_started":1}
I20260923 17:32:43.591315 25376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:43.590586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:34915)
0923 17:32:43.591062 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:43.591140 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":255,"threads_started":1}
I20260923 17:32:43.709257 25377 raft_consensus.cc:482] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:32:43.709456 25377 raft_consensus.cc:504] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:43.710091 25377 leader_election.cc:288] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395), 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
W20260923 17:32:43.711237 24983 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
W20260923 17:32:43.711431 24981 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
I20260923 17:32:43.711566 24981 leader_election.cc:302] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce71beefcc5e4c27a37c0200869b2d10; no voters: 5774c239fb4d4924be99efd11c8c84fc, d8616d2dab584f489b21e6bd3003031a
I20260923 17:32:43.711826 25377 raft_consensus.cc:2726] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:32:44.053273 25378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:44.052513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:37145)
0923 17:32:44.052983 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:44.053076 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":236,"threads_started":1}
I20260923 17:32:44.128553 25379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:44.127648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:60551)
0923 17:32:44.128228 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:44.128335 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":248,"threads_started":1}
I20260923 17:32:44.593029 25380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:44.592275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:37295)
0923 17:32:44.592749 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:44.592843 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":238,"threads_started":1}
I20260923 17:32:45.055160 25381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:45.054323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:53443)
0923 17:32:45.054898 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:45.054996 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":310,"threads_started":1}
I20260923 17:32:45.130307 25382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:45.129576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:57981)
0923 17:32:45.130048 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:45.130139 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":230,"threads_started":1}
I20260923 17:32:45.594821 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:45.594043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:39861)
0923 17:32:45.594554 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:45.594643 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":252,"threads_started":1}
I20260923 17:32:46.056803 25384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:46.056042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:51853)
0923 17:32:46.056517 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:46.056607 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":249,"threads_started":1}
I20260923 17:32:46.132157 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:46.131313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:56309)
0923 17:32:46.131784 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:46.131902 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":235,"threads_started":1}
I20260923 17:32:46.596583 25386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:46.595787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:41319)
0923 17:32:46.596307 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:46.596395 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":229,"threads_started":1}
I20260923 17:32:46.715497 25387 raft_consensus.cc:482] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:32:46.715683 25387 raft_consensus.cc:504] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:46.716368 25387 leader_election.cc:288] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395), 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
W20260923 17:32:46.717155 24983 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
W20260923 17:32:46.717320 24981 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
I20260923 17:32:46.717448 24981 leader_election.cc:302] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce71beefcc5e4c27a37c0200869b2d10; no voters: 5774c239fb4d4924be99efd11c8c84fc, d8616d2dab584f489b21e6bd3003031a
I20260923 17:32:46.717690 25387 raft_consensus.cc:2726] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:32:47.058610 25388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:47.057741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:44249)
0923 17:32:47.058239 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:47.058346 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":220,"threads_started":1}
I20260923 17:32:47.133948 25389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:47.133137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:57293)
0923 17:32:47.133618 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:47.133719 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":239,"threads_started":1}
I20260923 17:32:47.598407 25390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:47.597597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:42005)
0923 17:32:47.598109 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:47.598201 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":272,"threads_started":1}
I20260923 17:32:48.060329 25391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:48.059569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:48675)
0923 17:32:48.060050 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:48.060140 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":242,"threads_started":1}
I20260923 17:32:48.135751 25392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:48.134990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:52927)
0923 17:32:48.135477 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:48.135569 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":244,"threads_started":1}
I20260923 17:32:48.600310 25393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:48.599418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:47417)
0923 17:32:48.599993 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:48.600107 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":409,"thread_start_us":283,"threads_started":1}
I20260923 17:32:49.062011 25394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:49.061210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:56879)
0923 17:32:49.061740 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:49.061839 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":242,"threads_started":1}
I20260923 17:32:49.137530 25395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:49.136778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:48489)
0923 17:32:49.137255 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:49.137342 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":249,"threads_started":1}
I20260923 17:32:49.601905 25396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:49.601170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:43757)
0923 17:32:49.601640 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:49.601725 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":255,"threads_started":1}
I20260923 17:32:50.063681 25397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:50.062955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:47165)
0923 17:32:50.063424 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:50.063508 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":244,"threads_started":1}
I20260923 17:32:50.139403 25398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:50.138652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:47653)
0923 17:32:50.139089 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:50.139187 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":215,"threads_started":1}
I20260923 17:32:50.603729 25399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:50.602837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:49285)
0923 17:32:50.603471 (+ 634us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:50.603555 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":264,"threads_started":1}
I20260923 17:32:51.065368 25400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:51.064625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:53283)
0923 17:32:51.065119 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:51.065207 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":242,"threads_started":1}
I20260923 17:32:51.141235 25401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:51.140467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:41229)
0923 17:32:51.140954 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:51.141043 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":256,"threads_started":1}
I20260923 17:32:51.605466 25402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:51.604636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:59663)
0923 17:32:51.605131 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:51.605235 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20260923 17:32:52.067054 25403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:52.066256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:49205)
0923 17:32:52.066782 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:52.066872 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":292,"threads_started":1}
I20260923 17:32:52.143034 25404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:52.142177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:57457)
0923 17:32:52.142749 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:52.142836 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":333,"threads_started":1}
I20260923 17:32:52.607084 25405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:52.606334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:56407)
0923 17:32:52.606828 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:52.606919 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":261,"threads_started":1}
I20260923 17:32:52.883764 25406 raft_consensus.cc:482] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:32:52.883996 25406 raft_consensus.cc:504] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce71beefcc5e4c27a37c0200869b2d10" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 37087 } } peers { permanent_uuid: "d8616d2dab584f489b21e6bd3003031a" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43395 } } peers { permanent_uuid: "5774c239fb4d4924be99efd11c8c84fc" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 45417 } }
I20260923 17:32:52.884690 25406 leader_election.cc:288] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395), 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417)
W20260923 17:32:52.885730 24983 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8616d2dab584f489b21e6bd3003031a (127.8.29.194:43395): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
W20260923 17:32:52.885918 24981 leader_election.cc:341] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5774c239fb4d4924be99efd11c8c84fc (127.8.29.195:45417): Not found: Tablet not found: 42bcb16875304deab15cdbb8c42dbf33
I20260923 17:32:52.886065 24981 leader_election.cc:302] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ce71beefcc5e4c27a37c0200869b2d10; no voters: 5774c239fb4d4924be99efd11c8c84fc, d8616d2dab584f489b21e6bd3003031a
I20260923 17:32:52.886333 25406 raft_consensus.cc:2726] T 42bcb16875304deab15cdbb8c42dbf33 P ce71beefcc5e4c27a37c0200869b2d10 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 17:32:53.068718 25407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:53.068002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:56797)
0923 17:32:53.068465 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:53.068555 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":219,"threads_started":1}
I20260923 17:32:53.145068 25408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:53.144031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:43973)
0923 17:32:53.144584 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:53.144755 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":248,"threads_started":1}
I20260923 17:32:53.608737 25409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:53.607985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:49589)
0923 17:32:53.608446 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:53.608525 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":235,"threads_started":1}
I20260923 17:32:54.070705 25410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:54.069835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:45495)
0923 17:32:54.070385 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:54.070509 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":302,"threads_started":1}
I20260923 17:32:54.147289 25411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:54.146520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.193:45401)
0923 17:32:54.147014 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:54.147100 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":256,"threads_started":1}
I20260923 17:32:54.610896 25412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:54.609939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.194:45197)
0923 17:32:54.610539 (+ 600us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:54.610724 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":291,"threads_started":1}
I20260923 17:32:55.051936 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 24924
I20260923 17:32:55.067173 25413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:55.053052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36498 (local address 127.8.29.193:37087)
0923 17:32:55.053659 (+ 607us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:55.053667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:55.053693 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:32:55.053756 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:55.053762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:55.053962 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:55.054100 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:55.055575 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:55.056576 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:55.057834 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:55.058402 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:55.058550 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:55.061152 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:55.061165 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:55.061182 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:55.061225 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:55.064221 (+ 2996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:55.065043 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:55.065051 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:55.065053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:55.065203 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:55.065569 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:55.065572 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:55.065574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:55.066009 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:55.066245 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:55.066643 (+ 398us) server_negotiation.cc:299] Negotiation successful
0923 17:32:55.066897 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":324,"threads_started":1}
I20260923 17:32:55.068452 25089 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:55.072808 25415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:55.071967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:39585 (local address 127.8.29.195:44687)
0923 17:32:55.072503 (+ 536us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:55.072643 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.254:39585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":271,"threads_started":1}
I20260923 17:32:55.159154 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 24924
I20260923 17:32:55.173475 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 25098
I20260923 17:32:55.186247 25417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:55.174196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.8.29.194:43395)
0923 17:32:55.174640 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:55.174645 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:55.174661 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:32:55.174715 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:55.174719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:55.174784 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:55.174875 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:55.176204 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:55.176961 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:55.178108 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:55.178608 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:55.178716 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:55.181288 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:55.181293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:55.181296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:55.181330 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:55.183884 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:55.184499 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:55.184503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:55.184506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:55.184638 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:55.184948 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:55.184952 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:55.184954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:55.185232 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:55.185434 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:55.185735 (+ 301us) server_negotiation.cc:299] Negotiation successful
0923 17:32:55.185927 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":242,"threads_started":1}
I20260923 17:32:55.186934 25221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:55.268697 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 25098
I20260923 17:32:55.283972 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 25229
I20260923 17:32:55.297835 25419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:55.284843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51040 (local address 127.8.29.195:45417)
0923 17:32:55.285402 (+ 559us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:55.285407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:55.285422 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:32:55.285476 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:55.285480 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:55.285546 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:55.285641 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:55.287085 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:55.287897 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:55.289176 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:55.289604 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:55.289699 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:55.292288 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:55.292293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:55.292297 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:55.292333 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:55.295163 (+ 2830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:55.295880 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:55.295885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:55.295887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:55.296031 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:55.296429 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:55.296434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:55.296436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:55.296740 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:55.296968 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:55.297327 (+ 359us) server_negotiation.cc:299] Negotiation successful
0923 17:32:55.297508 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":283,"threads_started":1}
I20260923 17:32:55.298645 25352 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:55.381343 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 25229
2026-09-23T17:32:55Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64104 ms)
[----------] 30 tests from SecurityITest (296867 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:32:55 dist-test-slave-p7jt krb5kdc[25423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:32:55 dist-test-slave-p7jt krb5kdc[25423](info): set up 2 sockets
Sep 23 17:32:55 dist-test-slave-p7jt krb5kdc[25423](info): commencing operation
krb5kdc: starting...
W20260923 17:32:56.402618 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.984s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:32:56 dist-test-slave-p7jt krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:32:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:32:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:32:56.658704 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:41383
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:41383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:32:56.988242 25436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:56.988535 25436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:56.988657 25436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:56.996081 25436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:32:56.996205 25436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:56.996261 25436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:56.996351 25436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:32:56.996414 25436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:32:57.006546 25436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:41383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:41383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:57.007929 25436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:57.009225 25436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:57.021281 25441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:57.021234 25442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:57.021206 25444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:57.022655 25436 server_base.cc:1034] running on GCE node
I20260923 17:32:57.023489 25436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:57.024999 25436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:57.026212 25436 hybrid_clock.cc:648] HybridClock initialized: now 1790184777026169 us; error 58 us; skew 500 ppm
I20260923 17:32:57.026609 25436 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:57 dist-test-slave-p7jt krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184777, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:57.044068 25436 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:32:57.045781 25436 webserver.cc:460] Webserver started at http://127.8.29.254:39373/ using document root <none> and password file <none>
I20260923 17:32:57.046546 25436 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:57.046669 25436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:57.046984 25436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:32:57.050004 25436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "4db5840c18f447a6a78b67e3db4eb3f4"
format_stamp: "Formatted at 2026-09-23 17:32:57 on dist-test-slave-p7jt"
server_key: "22b80b42c38145b29e25552d616198e2"
server_key_iv: "efff8c59067ada3df904698c88d53a55"
server_key_version: "encryptionkey@0"
I20260923 17:32:57.050765 25436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "4db5840c18f447a6a78b67e3db4eb3f4"
format_stamp: "Formatted at 2026-09-23 17:32:57 on dist-test-slave-p7jt"
server_key: "22b80b42c38145b29e25552d616198e2"
server_key_iv: "efff8c59067ada3df904698c88d53a55"
server_key_version: "encryptionkey@0"
I20260923 17:32:57.055927 25436 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:32:57.059439 25451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:57.060832 25436 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:32:57.060988 25436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "4db5840c18f447a6a78b67e3db4eb3f4"
format_stamp: "Formatted at 2026-09-23 17:32:57 on dist-test-slave-p7jt"
server_key: "22b80b42c38145b29e25552d616198e2"
server_key_iv: "efff8c59067ada3df904698c88d53a55"
server_key_version: "encryptionkey@0"
I20260923 17:32:57.061165 25436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:57.077970 25436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:57.082193 25436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:57.082496 25436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:57.100083 25436 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:41383
I20260923 17:32:57.100127 25502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:41383 every 8 connection(s)
I20260923 17:32:57.101473 25436 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:32:57.103032 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 25436
I20260923 17:32:57.103256 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:32:57.104034 8311 external_mini_cluster.cc:1428] Setting key 08922168e9ab6f98b40f7f074b4bb2c8
I20260923 17:32:57.106225 25503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:32:57 dist-test-slave-p7jt krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:32:57.119655 25503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4: Bootstrap starting.
I20260923 17:32:57.123204 25503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4: Neither blocks nor log segments found. Creating new log.
I20260923 17:32:57.124269 25503 log.cc:824] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:57.127210 25503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4: No bootstrap required, opened a new log
I20260923 17:32:57.127543 25506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:57.106057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.8.29.254:41383)
0923 17:32:57.106850 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:57.106863 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:57.106904 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:32:57.108520 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:57.108541 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:57.108934 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:57.109258 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:57.111035 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:57.112436 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:57.114320 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:57.115121 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:57.115256 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:57.120139 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:57.120147 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:57.120164 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:57.120205 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:57.123545 (+ 3340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:57.124247 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:57.124258 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:57.124266 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:57.124429 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:57.124815 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:57.124819 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:57.124821 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:57.125370 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:57.125638 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:57.126078 (+ 440us) server_negotiation.cc:299] Negotiation successful
0923 17:32:57.126535 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":585,"thread_start_us":394,"threads_started":1}
I20260923 17:32:57.135155 25503 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5840c18f447a6a78b67e3db4eb3f4" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41383 } }
I20260923 17:32:57.135569 25503 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:32:57.135663 25503 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db5840c18f447a6a78b67e3db4eb3f4, State: Initialized, Role: FOLLOWER
I20260923 17:32:57.136243 25503 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5840c18f447a6a78b67e3db4eb3f4" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41383 } }
I20260923 17:32:57.136464 25503 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:32:57.136592 25503 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:32:57.136818 25503 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:32:57.139321 25503 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5840c18f447a6a78b67e3db4eb3f4" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41383 } }
I20260923 17:32:57.139817 25503 leader_election.cc:302] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4db5840c18f447a6a78b67e3db4eb3f4; no voters:
I20260923 17:32:57.140413 25503 leader_election.cc:288] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:32:57.140537 25508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:32:57.141182 25508 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [term 1 LEADER]: Becoming Leader. State: Replica: 4db5840c18f447a6a78b67e3db4eb3f4, State: Running, Role: LEADER
I20260923 17:32:57.141706 25508 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5840c18f447a6a78b67e3db4eb3f4" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41383 } }
I20260923 17:32:57.142369 25503 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:32:57.144320 25510 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db5840c18f447a6a78b67e3db4eb3f4. Latest consensus state: current_term: 1 leader_uuid: "4db5840c18f447a6a78b67e3db4eb3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5840c18f447a6a78b67e3db4eb3f4" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41383 } } }
I20260923 17:32:57.144769 25510 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:32:57.144345 25509 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4db5840c18f447a6a78b67e3db4eb3f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4db5840c18f447a6a78b67e3db4eb3f4" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 41383 } } }
I20260923 17:32:57.147825 25509 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4 [sys.catalog]: This master's current role is: LEADER
I20260923 17:32:57.148545 25514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:32:57.152774 25514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:32:57.161722 25514 catalog_manager.cc:1347] Generated new cluster ID: e18d61dbd7c9479f940d9c9ab6e904b0
I20260923 17:32:57.161851 25514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:32:57.173913 25514 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:32:57.175773 25514 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:32:57.182009 25514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4: Generated new TSK 0
I20260923 17:32:57.182821 25514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:32:57.250005 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41383
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:32:57.582084 25529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:57.582367 25529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:57.582482 25529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:57.589815 25529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:57.589936 25529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:57.590051 25529 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:32:57.600095 25529 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:57.601344 25529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:57.602730 25529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:57.616546 25534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:57.616573 25537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:57.616479 25535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:57.617465 25529 server_base.cc:1034] running on GCE node
I20260923 17:32:57.618088 25529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:57.619128 25529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:57.620343 25529 hybrid_clock.cc:648] HybridClock initialized: now 1790184777620279 us; error 57 us; skew 500 ppm
I20260923 17:32:57.620683 25529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:57 dist-test-slave-p7jt krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184777, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:57.623930 25529 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:32:57.625429 25529 webserver.cc:460] Webserver started at http://127.8.29.193:38091/ using document root <none> and password file <none>
I20260923 17:32:57.626106 25529 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:57.626209 25529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:57.626576 25529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:32:57.629595 25529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "eec64edc6d3f4c18ab5fa75cf311967b"
format_stamp: "Formatted at 2026-09-23 17:32:57 on dist-test-slave-p7jt"
server_key: "9fa044913ff74956b0617cf0c8636a0c"
server_key_iv: "ac41506f7e256d8460daeeb53d5cd36a"
server_key_version: "encryptionkey@0"
I20260923 17:32:57.630345 25529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "eec64edc6d3f4c18ab5fa75cf311967b"
format_stamp: "Formatted at 2026-09-23 17:32:57 on dist-test-slave-p7jt"
server_key: "9fa044913ff74956b0617cf0c8636a0c"
server_key_iv: "ac41506f7e256d8460daeeb53d5cd36a"
server_key_version: "encryptionkey@0"
I20260923 17:32:57.635818 25529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:32:57.639451 25544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:57.640904 25529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:32:57.641060 25529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "eec64edc6d3f4c18ab5fa75cf311967b"
format_stamp: "Formatted at 2026-09-23 17:32:57 on dist-test-slave-p7jt"
server_key: "9fa044913ff74956b0617cf0c8636a0c"
server_key_iv: "ac41506f7e256d8460daeeb53d5cd36a"
server_key_version: "encryptionkey@0"
I20260923 17:32:57.641247 25529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:57.652475 25529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:57.656914 25529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:57.657178 25529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:57.658154 25529 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:32:57.659515 25529 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:32:57.659590 25529 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:57.659672 25529 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:32:57.659714 25529 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:57.690315 25529 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:42577
I20260923 17:32:57.690385 25656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:42577 every 8 connection(s)
I20260923 17:32:57.691614 25529 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:32:57.693691 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 25529
I20260923 17:32:57.693910 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:32:57.694761 8311 external_mini_cluster.cc:1428] Setting key b58a6ebb15dd637c9a4b56dae2494026
Sep 23 17:32:57 dist-test-slave-p7jt krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184777, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:32:57.712656 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:57.694493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:40269 (local address 127.8.29.254:41383)
0923 17:32:57.695037 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:57.695042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:57.696205 (+ 1163us) server_negotiation.cc:407] Connection header received
0923 17:32:57.698282 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:57.698288 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:57.698380 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:57.698496 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:57.700070 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:57.700786 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:57.702417 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:57.702866 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:57.702981 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:57.706908 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:57.706914 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:57.706917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:57.706950 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:57.709265 (+ 2315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:57.710181 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:57.710186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:57.710188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:57.710335 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:57.710867 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:57.710872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:57.710874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:57.711179 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:57.711302 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:57.712176 (+ 874us) server_negotiation.cc:299] Negotiation successful
0923 17:32:57.712368 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":274,"threads_started":1}
I20260923 17:32:57.714026 25660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:57.695160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:41383 (local address 127.8.29.193:40269)
0923 17:32:57.696030 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:57.696089 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:57.698008 (+ 1919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:57.698761 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:57.698771 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:57.699305 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:32:57.699899 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:57.699919 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:57.700934 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:57.700939 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:57.702276 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:57.702290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:57.703094 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:57.703101 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:57.703333 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:57.704027 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:32:57.704056 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:32:57.706655 (+ 2599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:32:57.709414 (+ 2759us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:57.709424 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:57.709443 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:57.710039 (+ 596us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:57.710437 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:57.710442 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:57.710445 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:57.710710 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:57.711306 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:32:57.711314 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:32:57.711737 (+ 423us) client_negotiation.cc:769] Sending connection context
0923 17:32:57.712040 (+ 303us) client_negotiation.cc:240] Negotiation successful
0923 17:32:57.712622 (+ 582us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":343,"threads_started":1}
I20260923 17:32:57.715854 25657 heartbeater.cc:347] Connected to a master server at 127.8.29.254:41383
I20260923 17:32:57.716176 25657 heartbeater.cc:464] Registering TS with master...
I20260923 17:32:57.716894 25657 heartbeater.cc:511] Master 127.8.29.254:41383 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
I20260923 17:32:57.719288 25468 ts_manager.cc:194] Registered new tserver with Master: eec64edc6d3f4c18ab5fa75cf311967b (127.8.29.193:42577)
I20260923 17:32:57.720736 25468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:40269
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:32:57.751127 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41383
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:32:58.071036 25663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:58.071328 25663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:58.071424 25663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:58.078905 25663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:58.079034 25663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:58.079149 25663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:32:58.089092 25663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:58.090348 25663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:58.091610 25663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:58.104933 25669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:58.105026 25668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:58.105536 25671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:58.107049 25663 server_base.cc:1034] running on GCE node
I20260923 17:32:58.107672 25663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:58.108632 25663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:58.109858 25663 hybrid_clock.cc:648] HybridClock initialized: now 1790184778109805 us; error 37 us; skew 500 ppm
I20260923 17:32:58.110176 25663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:58 dist-test-slave-p7jt krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184778, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:58.113485 25663 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:32:58.115053 25663 webserver.cc:460] Webserver started at http://127.8.29.194:42827/ using document root <none> and password file <none>
I20260923 17:32:58.115751 25663 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:58.115864 25663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:58.116158 25663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:32:58.119165 25663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "b81201a3d25640eeaa3410c120e2bbc4"
format_stamp: "Formatted at 2026-09-23 17:32:58 on dist-test-slave-p7jt"
server_key: "72855b02526da1f34d556e9afc3dcccc"
server_key_iv: "d241134ee873c6c4f5f5ebad5dafae31"
server_key_version: "encryptionkey@0"
I20260923 17:32:58.119972 25663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "b81201a3d25640eeaa3410c120e2bbc4"
format_stamp: "Formatted at 2026-09-23 17:32:58 on dist-test-slave-p7jt"
server_key: "72855b02526da1f34d556e9afc3dcccc"
server_key_iv: "d241134ee873c6c4f5f5ebad5dafae31"
server_key_version: "encryptionkey@0"
I20260923 17:32:58.125317 25663 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260923 17:32:58.129014 25678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:58.130452 25663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:32:58.130657 25663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "b81201a3d25640eeaa3410c120e2bbc4"
format_stamp: "Formatted at 2026-09-23 17:32:58 on dist-test-slave-p7jt"
server_key: "72855b02526da1f34d556e9afc3dcccc"
server_key_iv: "d241134ee873c6c4f5f5ebad5dafae31"
server_key_version: "encryptionkey@0"
I20260923 17:32:58.130864 25663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:58.149456 25663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:58.153872 25663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:58.154150 25663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:58.155208 25663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:32:58.156633 25663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:32:58.156721 25663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:58.156839 25663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:32:58.156888 25663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:58.186733 25663 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:37179
I20260923 17:32:58.186808 25790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:37179 every 8 connection(s)
I20260923 17:32:58.187978 25663 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:32:58.194895 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 25663
I20260923 17:32:58.195058 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:32:58.195497 8311 external_mini_cluster.cc:1428] Setting key 58af712878478bd9677f44b0d617e6e6
Sep 23 17:32:58 dist-test-slave-p7jt krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184778, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:32:58.207784 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.190561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:58867 (local address 127.8.29.254:41383)
0923 17:32:58.190749 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:58.190754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:58.191972 (+ 1218us) server_negotiation.cc:407] Connection header received
0923 17:32:58.193880 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:58.193886 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:58.193971 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:58.194057 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:58.195750 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.196485 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.198041 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.198514 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.198628 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.202002 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:58.202007 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:58.202011 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:58.202045 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:58.204584 (+ 2539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:58.205399 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:58.205405 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:58.205407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:58.205542 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:58.205989 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:58.205993 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:58.205995 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:58.206300 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:58.206450 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:58.207198 (+ 748us) server_negotiation.cc:299] Negotiation successful
0923 17:32:58.207455 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:32:58.208839 25793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.191050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:41383 (local address 127.8.29.194:58867)
0923 17:32:58.191801 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:58.191840 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:58.193591 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:58.194312 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:58.194321 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:58.194947 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:32:58.195576 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.195601 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.196634 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.196638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.197900 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.197913 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.198743 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.198752 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.198976 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.199604 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:32:58.199631 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:32:58.201762 (+ 2131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:32:58.204794 (+ 3032us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:58.204802 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:58.204817 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:58.205267 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:58.205656 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:58.205661 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:58.205664 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:58.205877 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:58.206428 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:32:58.206436 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:32:58.206832 (+ 396us) client_negotiation.cc:769] Sending connection context
0923 17:32:58.207092 (+ 260us) client_negotiation.cc:240] Negotiation successful
0923 17:32:58.207663 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"thread_start_us":308,"threads_started":1}
I20260923 17:32:58.210408 25791 heartbeater.cc:347] Connected to a master server at 127.8.29.254:41383
I20260923 17:32:58.210783 25791 heartbeater.cc:464] Registering TS with master...
I20260923 17:32:58.211666 25791 heartbeater.cc:511] Master 127.8.29.254:41383 requested a full tablet report, sending...
I20260923 17:32:58.213305 25468 ts_manager.cc:194] Registered new tserver with Master: b81201a3d25640eeaa3410c120e2bbc4 (127.8.29.194:37179)
I20260923 17:32:58.214391 25468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:58867
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:32:58.249567 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41383
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:32:58.575938 25796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:32:58.576233 25796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:32:58.576355 25796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:32:58.583746 25796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:32:58.583868 25796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:32:58.584023 25796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:32:58.594120 25796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:41383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:32:58.595574 25796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:32:58.596875 25796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:32:58.610577 25802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:58.610610 25801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:32:58.610644 25804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:32:58.611424 25796 server_base.cc:1034] running on GCE node
I20260923 17:32:58.612049 25796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:32:58.613092 25796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:32:58.614295 25796 hybrid_clock.cc:648] HybridClock initialized: now 1790184778614246 us; error 34 us; skew 500 ppm
I20260923 17:32:58.614629 25796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:32:58 dist-test-slave-p7jt krb5kdc[25423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184778, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:32:58.617578 25796 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:32:58.619123 25796 webserver.cc:460] Webserver started at http://127.8.29.195:37251/ using document root <none> and password file <none>
I20260923 17:32:58.619817 25796 fs_manager.cc:359] Metadata directory not provided
I20260923 17:32:58.619925 25796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:32:58.620219 25796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:32:58.623138 25796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "995eaf96aa974dfe9319fcf0f86937d1"
format_stamp: "Formatted at 2026-09-23 17:32:58 on dist-test-slave-p7jt"
server_key: "0a287c39899f8efe3edc83e5ec9f78b1"
server_key_iv: "9c32e3d872fac2384d78a89ec5ee07b3"
server_key_version: "encryptionkey@0"
I20260923 17:32:58.623916 25796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "995eaf96aa974dfe9319fcf0f86937d1"
format_stamp: "Formatted at 2026-09-23 17:32:58 on dist-test-slave-p7jt"
server_key: "0a287c39899f8efe3edc83e5ec9f78b1"
server_key_iv: "9c32e3d872fac2384d78a89ec5ee07b3"
server_key_version: "encryptionkey@0"
I20260923 17:32:58.629164 25796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:32:58.632705 25811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:58.634305 25796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:32:58.634511 25796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "995eaf96aa974dfe9319fcf0f86937d1"
format_stamp: "Formatted at 2026-09-23 17:32:58 on dist-test-slave-p7jt"
server_key: "0a287c39899f8efe3edc83e5ec9f78b1"
server_key_iv: "9c32e3d872fac2384d78a89ec5ee07b3"
server_key_version: "encryptionkey@0"
I20260923 17:32:58.634728 25796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:32:58.649339 25796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:32:58.653796 25796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:32:58.654071 25796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:32:58.655125 25796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:32:58.656487 25796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:32:58.656556 25796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:58.656652 25796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:32:58.656720 25796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:32:58.686875 25796 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:46519
I20260923 17:32:58.686919 25923 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:46519 every 8 connection(s)
I20260923 17:32:58.688215 25796 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:32:58.695715 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 25796
I20260923 17:32:58.695926 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:32:58.696483 8311 external_mini_cluster.cc:1428] Setting key 20025613a3b5a4d414f6a9cfc6b5529b
Sep 23 17:32:58 dist-test-slave-p7jt krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184778, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:32:58.709390 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.691068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:56827 (local address 127.8.29.254:41383)
0923 17:32:58.691255 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:58.691260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:58.692914 (+ 1654us) server_negotiation.cc:407] Connection header received
0923 17:32:58.694813 (+ 1899us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:58.694820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:58.694938 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:58.695031 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:58.697322 (+ 2291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.698295 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.700143 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.700533 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.700635 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.704143 (+ 3508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:58.704148 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:58.704152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:58.704187 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:58.706637 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:58.707345 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:58.707348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:58.707351 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:58.707472 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:58.707892 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:58.707895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:58.707897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:58.708168 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:58.708285 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:58.708960 (+ 675us) server_negotiation.cc:299] Negotiation successful
0923 17:32:58.709126 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:32:58.710557 25926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.691763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:41383 (local address 127.8.29.195:56827)
0923 17:32:58.692702 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:58.692753 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:58.694486 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:58.695335 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:58.695347 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:58.696171 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:32:58.697103 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.697129 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.698524 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.698530 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.700015 (+ 1485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.700028 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.700741 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.700745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.700959 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.701632 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:32:58.701656 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:32:58.703930 (+ 2274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:32:58.706779 (+ 2849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:58.706787 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:58.706802 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:58.707245 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:58.707570 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:32:58.707574 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:32:58.707576 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:32:58.707800 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:32:58.708279 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:32:58.708286 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:32:58.708621 (+ 335us) client_negotiation.cc:769] Sending connection context
0923 17:32:58.708884 (+ 263us) client_negotiation.cc:240] Negotiation successful
0923 17:32:58.709273 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":396,"threads_started":1}
I20260923 17:32:58.711947 25924 heartbeater.cc:347] Connected to a master server at 127.8.29.254:41383
I20260923 17:32:58.712284 25924 heartbeater.cc:464] Registering TS with master...
I20260923 17:32:58.713021 25924 heartbeater.cc:511] Master 127.8.29.254:41383 requested a full tablet report, sending...
I20260923 17:32:58.714483 25468 ts_manager.cc:194] Registered new tserver with Master: 995eaf96aa974dfe9319fcf0f86937d1 (127.8.29.195:46519)
I20260923 17:32:58.715294 25468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:56827
I20260923 17:32:58.724386 25657 heartbeater.cc:503] Master 127.8.29.254:41383 was elected leader, sending a full tablet report...
I20260923 17:32:58.725180 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:32:58.741865 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.729769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54028 (local address 127.8.29.254:41383)
0923 17:32:58.729930 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:58.729935 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:58.730342 (+ 407us) server_negotiation.cc:407] Connection header received
0923 17:32:58.730518 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:58.730522 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:58.730587 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:58.730661 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:58.732080 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.732823 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.734237 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.734679 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.734786 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.737078 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:58.737084 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:58.737087 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:58.737117 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:58.739497 (+ 2380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:58.740205 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:58.740209 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:58.740212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:58.740349 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:58.740664 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:58.740667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:58.740669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:58.740962 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:58.741139 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:58.741472 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 17:32:58.741631 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:32:58.746055 25468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:32:58.752918 25468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:32:58.775909 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.765005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:37179 (local address 127.0.0.1:33720)
0923 17:32:58.766217 (+ 1212us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:58.766274 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:58.766556 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:58.767620 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:58.767628 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:58.767660 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:58.768061 (+ 401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.768074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.770094 (+ 2020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.770100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.772489 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.772506 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.774866 (+ 2360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.774873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.775114 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.775169 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 17:32:58.775306 (+ 137us) client_negotiation.cc:240] Negotiation successful
0923 17:32:58.775521 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":889,"thread_start_us":695,"threads_started":1}
I20260923 17:32:58.776829 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.765031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.8.29.194:37179)
0923 17:32:58.765567 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:58.765577 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:58.766368 (+ 791us) server_negotiation.cc:407] Connection header received
0923 17:32:58.766842 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:58.766852 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:58.767127 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:58.767298 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:58.768237 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.769898 (+ 1661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.772676 (+ 2778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.774277 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.774393 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.774648 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:58.776308 (+ 1660us) server_negotiation.cc:299] Negotiation successful
0923 17:32:58.776478 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":391,"thread_start_us":279,"threads_started":1}
I20260923 17:32:58.783505 25726 tablet_service.cc:1467] Processing CreateTablet for tablet 4195635f39884007bdd0bdf63cc80f00 (DEFAULT_TABLE table=test-table [id=e0df3601a87a473d87bc9a55bb2510a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:32:58.784960 25726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4195635f39884007bdd0bdf63cc80f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:32:58.786610 25936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.766409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42577 (local address 127.0.0.1:35756)
0923 17:32:58.777465 (+ 11056us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:58.777486 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:58.777599 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:58.779799 (+ 2200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:58.779804 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:58.779829 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:58.780016 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.780027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.782687 (+ 2660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.782692 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.784485 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.784496 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.785761 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.785767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.785935 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.785969 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:32:58.786057 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:32:58.786232 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10947,"mutex_wait_us":21,"thread_start_us":12491,"threads_started":1}
I20260923 17:32:58.787014 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.766625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.8.29.193:42577)
0923 17:32:58.778822 (+ 12197us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:58.778831 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:58.778862 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:32:58.778964 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:58.778973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:58.779237 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:58.779808 (+ 571us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:58.780184 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.782541 (+ 2357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.784625 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.785602 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.785725 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.785967 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:58.786112 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 17:32:58.786268 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12003,"thread_start_us":11868,"threads_started":1}
I20260923 17:32:58.787801 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.767188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47968 (local address 127.8.29.195:46519)
0923 17:32:58.778259 (+ 11071us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:58.778269 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:58.779352 (+ 1083us) server_negotiation.cc:407] Connection header received
0923 17:32:58.779543 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:58.779551 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:58.779803 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:58.779984 (+ 181us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:58.780744 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.782031 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.784326 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.785678 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.785801 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.786011 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:58.787353 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0923 17:32:58.787511 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10875,"thread_start_us":10699,"threads_started":1}
I20260923 17:32:58.788260 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:58.766641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:46519 (local address 127.0.0.1:47968)
0923 17:32:58.779197 (+ 12556us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:58.779218 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:58.779364 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:58.780062 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:58.780066 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:58.780090 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:58.780385 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.780394 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.782163 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.782168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.784150 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:58.784165 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:58.786156 (+ 1991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:58.786163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:58.786354 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:58.786386 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:32:58.786506 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 17:32:58.786648 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12420,"mutex_wait_us":26,"thread_start_us":12880,"threads_started":1}
I20260923 17:32:58.790566 25592 tablet_service.cc:1467] Processing CreateTablet for tablet 4195635f39884007bdd0bdf63cc80f00 (DEFAULT_TABLE table=test-table [id=e0df3601a87a473d87bc9a55bb2510a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:32:58.792076 25592 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4195635f39884007bdd0bdf63cc80f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:32:58.791846 25859 tablet_service.cc:1467] Processing CreateTablet for tablet 4195635f39884007bdd0bdf63cc80f00 (DEFAULT_TABLE table=test-table [id=e0df3601a87a473d87bc9a55bb2510a9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:32:58.793277 25859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4195635f39884007bdd0bdf63cc80f00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:32:58.797808 25942 tablet_bootstrap.cc:492] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: Bootstrap starting.
I20260923 17:32:58.800884 25942 tablet_bootstrap.cc:654] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: Neither blocks nor log segments found. Creating new log.
I20260923 17:32:58.801836 25942 log.cc:824] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:58.805167 25942 tablet_bootstrap.cc:492] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: No bootstrap required, opened a new log
I20260923 17:32:58.805428 25942 ts_tablet_manager.cc:1397] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260923 17:32:58.806830 25944 tablet_bootstrap.cc:492] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b: Bootstrap starting.
I20260923 17:32:58.808271 25945 tablet_bootstrap.cc:492] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: Bootstrap starting.
I20260923 17:32:58.810302 25944 tablet_bootstrap.cc:654] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b: Neither blocks nor log segments found. Creating new log.
I20260923 17:32:58.810927 25945 tablet_bootstrap.cc:654] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: Neither blocks nor log segments found. Creating new log.
I20260923 17:32:58.811512 25944 log.cc:824] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:58.811841 25945 log.cc:824] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: Log is configured to *not* fsync() on all Append() calls
I20260923 17:32:58.814574 25945 tablet_bootstrap.cc:492] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: No bootstrap required, opened a new log
I20260923 17:32:58.814795 25945 ts_tablet_manager.cc:1397] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260923 17:32:58.816051 25942 raft_consensus.cc:348] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } }
I20260923 17:32:58.816529 25942 raft_consensus.cc:374] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:32:58.816627 25942 raft_consensus.cc:729] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b81201a3d25640eeaa3410c120e2bbc4, State: Initialized, Role: FOLLOWER
I20260923 17:32:58.817296 25942 consensus_queue.cc:260] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } }
I20260923 17:32:58.818440 25791 heartbeater.cc:503] Master 127.8.29.254:41383 was elected leader, sending a full tablet report...
I20260923 17:32:58.819523 25942 ts_tablet_manager.cc:1428] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260923 17:32:58.820827 25944 tablet_bootstrap.cc:492] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b: No bootstrap required, opened a new log
I20260923 17:32:58.821108 25944 ts_tablet_manager.cc:1397] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20260923 17:32:58.822707 25945 raft_consensus.cc:348] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } }
I20260923 17:32:58.823143 25945 raft_consensus.cc:374] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:32:58.823244 25945 raft_consensus.cc:729] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 995eaf96aa974dfe9319fcf0f86937d1, State: Initialized, Role: FOLLOWER
I20260923 17:32:58.823807 25945 consensus_queue.cc:260] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } }
I20260923 17:32:58.824865 25924 heartbeater.cc:503] Master 127.8.29.254:41383 was elected leader, sending a full tablet report...
I20260923 17:32:58.826174 25945 ts_tablet_manager.cc:1428] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260923 17:32:58.832974 25944 raft_consensus.cc:348] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } }
I20260923 17:32:58.833482 25944 raft_consensus.cc:374] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:32:58.833575 25944 raft_consensus.cc:729] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eec64edc6d3f4c18ab5fa75cf311967b, State: Initialized, Role: FOLLOWER
I20260923 17:32:58.834661 25944 consensus_queue.cc:260] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } }
I20260923 17:32:58.836391 25944 ts_tablet_manager.cc:1428] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b: Time spent starting tablet: real 0.015s user 0.013s sys 0.000s
W20260923 17:32:58.939926 25925 tablet.cc:2369] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:32:58.939929 25792 tablet.cc:2369] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:32:58.943882 25658 tablet.cc:2369] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:32:59.007104 25950 raft_consensus.cc:482] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:32:59.007437 25950 raft_consensus.cc:504] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } }
I20260923 17:32:59.009112 25950 leader_election.cc:288] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b81201a3d25640eeaa3410c120e2bbc4 (127.8.29.194:37179), 995eaf96aa974dfe9319fcf0f86937d1 (127.8.29.195:46519)
I20260923 17:32:59.015467 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.009563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:41635 (local address 127.8.29.194:37179)
0923 17:32:59.009797 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:59.009804 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:59.009997 (+ 193us) server_negotiation.cc:407] Connection header received
0923 17:32:59.010236 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:59.010243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:59.010325 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:59.010416 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:59.011137 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.011835 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.013781 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.014581 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.014733 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:59.014809 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:59.015083 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 17:32:59.015208 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:32:59.015574 25951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.009340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:37179 (local address 127.8.29.193:41635)
0923 17:32:59.009888 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:59.009904 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:59.010068 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:59.010682 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:59.010686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:59.010712 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:59.010995 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:59.011002 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.011966 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.011976 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:59.013638 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:59.013650 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.014777 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.014783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:59.014938 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:59.014963 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:32:59.015047 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:32:59.015177 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"mutex_wait_us":22,"thread_start_us":273,"threads_started":1}
I20260923 17:32:59.015666 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.010290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:36003 (local address 127.8.29.195:46519)
0923 17:32:59.010616 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:59.010621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:59.010762 (+ 141us) server_negotiation.cc:407] Connection header received
0923 17:32:59.010885 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:59.010889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:59.010952 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:59.011022 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:32:59.011534 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.012245 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.013974 (+ 1729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.014784 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.014933 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:59.015007 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:59.015307 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 17:32:59.015431 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195}
I20260923 17:32:59.015935 25952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.010029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:46519 (local address 127.8.29.193:36003)
0923 17:32:59.010640 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:59.010653 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:32:59.010772 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:32:59.011054 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:32:59.011059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:32:59.011081 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:32:59.011393 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:59.011401 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.012379 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.012385 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:59.013824 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:32:59.013837 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.014994 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.015001 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:32:59.015151 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:59.015174 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:32:59.015262 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 17:32:59.015389 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":20,"thread_start_us":272,"threads_started":1}
I20260923 17:32:59.016566 25746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195635f39884007bdd0bdf63cc80f00" candidate_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81201a3d25640eeaa3410c120e2bbc4" is_pre_election: true
I20260923 17:32:59.016971 25879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195635f39884007bdd0bdf63cc80f00" candidate_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995eaf96aa974dfe9319fcf0f86937d1" is_pre_election: true
I20260923 17:32:59.017242 25746 raft_consensus.cc:2454] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eec64edc6d3f4c18ab5fa75cf311967b in term 0.
I20260923 17:32:59.017596 25879 raft_consensus.cc:2454] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eec64edc6d3f4c18ab5fa75cf311967b in term 0.
I20260923 17:32:59.018002 25546 leader_election.cc:302] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81201a3d25640eeaa3410c120e2bbc4, eec64edc6d3f4c18ab5fa75cf311967b; no voters:
I20260923 17:32:59.018399 25950 raft_consensus.cc:2781] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:32:59.018567 25950 raft_consensus.cc:482] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:32:59.018661 25950 raft_consensus.cc:3037] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:32:59.020776 25950 raft_consensus.cc:504] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } }
I20260923 17:32:59.021359 25950 leader_election.cc:288] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [CANDIDATE]: Term 1 election: Requested vote from peers b81201a3d25640eeaa3410c120e2bbc4 (127.8.29.194:37179), 995eaf96aa974dfe9319fcf0f86937d1 (127.8.29.195:46519)
I20260923 17:32:59.021885 25746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195635f39884007bdd0bdf63cc80f00" candidate_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b81201a3d25640eeaa3410c120e2bbc4"
I20260923 17:32:59.022038 25879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4195635f39884007bdd0bdf63cc80f00" candidate_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "995eaf96aa974dfe9319fcf0f86937d1"
I20260923 17:32:59.022146 25746 raft_consensus.cc:3037] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:32:59.022256 25879 raft_consensus.cc:3037] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:32:59.025394 25879 raft_consensus.cc:2454] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eec64edc6d3f4c18ab5fa75cf311967b in term 1.
I20260923 17:32:59.025394 25746 raft_consensus.cc:2454] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eec64edc6d3f4c18ab5fa75cf311967b in term 1.
I20260923 17:32:59.025890 25546 leader_election.cc:302] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b81201a3d25640eeaa3410c120e2bbc4, eec64edc6d3f4c18ab5fa75cf311967b; no voters:
I20260923 17:32:59.026201 25950 raft_consensus.cc:2781] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:32:59.026665 25950 raft_consensus.cc:686] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 1 LEADER]: Becoming Leader. State: Replica: eec64edc6d3f4c18ab5fa75cf311967b, State: Running, Role: LEADER
I20260923 17:32:59.027133 25950 consensus_queue.cc:237] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } }
I20260923 17:32:59.031996 25468 catalog_manager.cc:5495] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b reported cstate change: term changed from 0 to 1, leader changed from <none> to eec64edc6d3f4c18ab5fa75cf311967b (127.8.29.193). New cstate: current_term: 1 leader_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eec64edc6d3f4c18ab5fa75cf311967b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 42577 } health_report { overall_health: HEALTHY } } }
I20260923 17:32:59.070911 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.065414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35770 (local address 127.8.29.193:42577)
0923 17:32:59.065575 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:59.065579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:59.065648 (+ 69us) server_negotiation.cc:407] Connection header received
0923 17:32:59.065838 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:59.065843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:59.065946 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:59.066067 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:32:59.066401 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.067166 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.068826 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.069251 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.069370 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:59.069843 (+ 473us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:32:59.070384 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:32:59.070536 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:59.070631 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 17:32:59.070707 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:32:59.079219 25879 raft_consensus.cc:1261] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [term 1 FOLLOWER]: Refusing update from remote peer eec64edc6d3f4c18ab5fa75cf311967b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:32:59.079219 25746 raft_consensus.cc:1261] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [term 1 FOLLOWER]: Refusing update from remote peer eec64edc6d3f4c18ab5fa75cf311967b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:32:59.080144 25950 consensus_queue.cc:1035] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [LEADER]: Connected to new peer: Peer: permanent_uuid: "995eaf96aa974dfe9319fcf0f86937d1" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 46519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:32:59.080492 25953 consensus_queue.cc:1035] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b81201a3d25640eeaa3410c120e2bbc4" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 37179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:32:59.091544 25956 mvcc.cc:205] Tried to move back new op lower bound from 7332596855098048512 to 7332596854898970624. Current Snapshot: MvccSnapshot[applied={T|T < 7332596855098048512}]
I20260923 17:32:59.108134 25467 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:32:59.108440 25467 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:32:59.112121 25467 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4db5840c18f447a6a78b67e3db4eb3f4: Sending DeleteTablet for 3 replicas of tablet 4195635f39884007bdd0bdf63cc80f00
I20260923 17:32:59.113483 25592 tablet_service.cc:1514] Processing DeleteTablet for tablet 4195635f39884007bdd0bdf63cc80f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:32:59 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:35756
I20260923 17:32:59.113483 25726 tablet_service.cc:1514] Processing DeleteTablet for tablet 4195635f39884007bdd0bdf63cc80f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:32:59 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:33720
I20260923 17:32:59.113652 25859 tablet_service.cc:1514] Processing DeleteTablet for tablet 4195635f39884007bdd0bdf63cc80f00 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:32:59 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:47968
I20260923 17:32:59.116463 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 25529
I20260923 17:32:59.131574 25967 tablet_replica.cc:330] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b: stopping tablet replica
I20260923 17:32:59.133437 25967 raft_consensus.cc:2229] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:32:59.135175 25966 tablet_replica.cc:330] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: stopping tablet replica
I20260923 17:32:59.134202 25967 raft_consensus.cc:2258] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:32:59.141323 25966 raft_consensus.cc:2229] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:32:59.141898 25966 raft_consensus.cc:2258] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:32:59.144070 25968 tablet_replica.cc:330] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: stopping tablet replica
I20260923 17:32:59.144543 25968 raft_consensus.cc:2229] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:32:59.144919 25967 ts_tablet_manager.cc:1905] T 4195635f39884007bdd0bdf63cc80f00 P eec64edc6d3f4c18ab5fa75cf311967b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:32:59.145045 25968 raft_consensus.cc:2258] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:32:59.145339 25966 ts_tablet_manager.cc:1905] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 17:32:59 dist-test-slave-p7jt krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:32:59.147352 25968 ts_tablet_manager.cc:1905] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:32:59.152226 25966 ts_tablet_manager.cc:1918] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:32:59.152441 25966 log.cc:1196] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/4195635f39884007bdd0bdf63cc80f00
I20260923 17:32:59.152997 25966 ts_tablet_manager.cc:1939] T 4195635f39884007bdd0bdf63cc80f00 P b81201a3d25640eeaa3410c120e2bbc4: Deleting consensus metadata
I20260923 17:32:59.155035 25968 ts_tablet_manager.cc:1918] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:32:59.155083 25453 catalog_manager.cc:4841] TS b81201a3d25640eeaa3410c120e2bbc4 (127.8.29.194:37179): tablet 4195635f39884007bdd0bdf63cc80f00 (table test-table [id=e0df3601a87a473d87bc9a55bb2510a9]) successfully deleted
I20260923 17:32:59.155202 25968 log.cc:1196] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/4195635f39884007bdd0bdf63cc80f00
I20260923 17:32:59.155679 25940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.117793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.8.29.193:42577)
0923 17:32:59.117984 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:59.117991 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:59.132823 (+ 14832us) server_negotiation.cc:407] Connection header received
0923 17:32:59.133053 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:59.133059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:59.133159 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:59.133263 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:59.135620 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.136940 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.139017 (+ 2077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.139765 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.139874 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:59.146953 (+ 7079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:59.146970 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:59.146987 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:59.147038 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:59.151149 (+ 4111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:59.152232 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:59.152242 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:59.152246 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:59.152687 (+ 441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:59.153344 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:59.153349 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:59.153352 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:59.153835 (+ 483us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:59.154136 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:59.154843 (+ 707us) server_negotiation.cc:299] Negotiation successful
0923 17:32:59.155101 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:32:59.155679 25968 ts_tablet_manager.cc:1939] T 4195635f39884007bdd0bdf63cc80f00 P 995eaf96aa974dfe9319fcf0f86937d1: Deleting consensus metadata
I20260923 17:32:59.156836 25652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:59.157687 25453 catalog_manager.cc:4841] TS 995eaf96aa974dfe9319fcf0f86937d1 (127.8.29.195:46519): tablet 4195635f39884007bdd0bdf63cc80f00 (table test-table [id=e0df3601a87a473d87bc9a55bb2510a9]) successfully deleted
I20260923 17:32:59.266654 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 25529
I20260923 17:32:59.281390 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 25663
W20260923 17:32:59.281409 25455 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.29.193:42577 (error 108)
I20260923 17:32:59.283159 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.282675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42577 (local address 127.0.0.1:35784)
0923 17:32:59.282842 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:59.282981 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:32:59.283381 25455 catalog_manager.cc:4584] TS eec64edc6d3f4c18ab5fa75cf311967b (127.8.29.193:42577): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4195635f39884007bdd0bdf63cc80f00: Network error: Client connection negotiation failed: client connection to 127.8.29.193:42577: connect: Connection refused (error 111)
Sep 23 17:32:59 dist-test-slave-p7jt krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:32:59.296404 25937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.282176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33732 (local address 127.8.29.194:37179)
0923 17:32:59.282356 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:59.282363 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:59.282509 (+ 146us) server_negotiation.cc:407] Connection header received
0923 17:32:59.282722 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:59.282727 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:59.282828 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:59.282982 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:59.284338 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.285108 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.286405 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.286868 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.286975 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:59.291037 (+ 4062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:59.291053 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:59.291068 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:59.291110 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:59.293802 (+ 2692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:59.294566 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:59.294575 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:59.294577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:59.294707 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:59.295028 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:59.295031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:59.295033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:59.295423 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:59.295644 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:59.295948 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 17:32:59.296117 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:32:59.297101 25786 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:59.336820 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.336345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42577 (local address 127.0.0.1:35792)
0923 17:32:59.336555 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:59.336657 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:32:59.337932 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.337552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42577 (local address 127.0.0.1:35800)
0923 17:32:59.337691 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:59.337791 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 17:32:59.384922 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 25663
I20260923 17:32:59.398792 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.398203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42577 (local address 127.0.0.1:35814)
0923 17:32:59.398443 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:59.398579 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260923 17:32:59.399029 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 25796
I20260923 17:32:59.399827 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.399448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42577 (local address 127.0.0.1:35820)
0923 17:32:59.399603 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:59.399683 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
Sep 23 17:32:59 dist-test-slave-p7jt krb5kdc[25423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184776, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:32:59.413707 25941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.399765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47980 (local address 127.8.29.195:46519)
0923 17:32:59.399944 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:32:59.399950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:32:59.399970 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:32:59.400045 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:32:59.400050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:32:59.400118 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:32:59.400195 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:32:59.401440 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.402153 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.403424 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:32:59.403850 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:32:59.403959 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:32:59.408346 (+ 4387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:32:59.408363 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:32:59.408380 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:32:59.408420 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:32:59.411004 (+ 2584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:59.411691 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:59.411699 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:59.411701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:59.411839 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:32:59.412202 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:32:59.412206 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:32:59.412208 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:32:59.412574 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:32:59.412839 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0923 17:32:59.413160 (+ 321us) server_negotiation.cc:299] Negotiation successful
0923 17:32:59.413379 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:32:59.414350 25919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:59.480701 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.480178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42577 (local address 127.0.0.1:35822)
0923 17:32:59.480427 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:59.480527 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260923 17:32:59.481853 25938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:32:59.481397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:42577 (local address 127.0.0.1:35826)
0923 17:32:59.481631 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 17:32:59.481718 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260923 17:32:59.502496 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 25796
I20260923 17:32:59.515848 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 25436
I20260923 17:32:59.517035 25498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:32:59.627386 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 25436
2026-09-23T17:32:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4248 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T17:32:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:32:59Z Disabled control of system clock
I20260923 17:32:59.698611 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:38081
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:38081
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:33:00.030711 25978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:00.031092 25978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:00.031188 25978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:00.038616 25978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:00.038743 25978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:00.038789 25978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:00.038844 25978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:00.038887 25978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:00.048188 25978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:38081
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:38081
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:00.049456 25978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:00.051275 25978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:00.063076 25986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:00.063122 25984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:00.063056 25983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:00.064251 25978 server_base.cc:1034] running on GCE node
I20260923 17:33:00.065069 25978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:00.066529 25978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:00.067773 25978 hybrid_clock.cc:648] HybridClock initialized: now 1790184780067714 us; error 49 us; skew 500 ppm
I20260923 17:33:00.068146 25978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:00.070727 25978 webserver.cc:460] Webserver started at http://127.8.29.254:33477/ using document root <none> and password file <none>
I20260923 17:33:00.071457 25978 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:00.071573 25978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:00.071888 25978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:00.075246 25978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "4701be8736f5445d824598098624bfaf"
format_stamp: "Formatted at 2026-09-23 17:33:00 on dist-test-slave-p7jt"
server_key: "a56204c0471e9ffa0a1d571fa09e7deb"
server_key_iv: "415a44487c9c5770e52d322078f8b9fd"
server_key_version: "encryptionkey@0"
I20260923 17:33:00.076031 25978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "4701be8736f5445d824598098624bfaf"
format_stamp: "Formatted at 2026-09-23 17:33:00 on dist-test-slave-p7jt"
server_key: "a56204c0471e9ffa0a1d571fa09e7deb"
server_key_iv: "415a44487c9c5770e52d322078f8b9fd"
server_key_version: "encryptionkey@0"
I20260923 17:33:00.081332 25978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:33:00.085348 25992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:00.086977 25978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:00.087174 25978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "4701be8736f5445d824598098624bfaf"
format_stamp: "Formatted at 2026-09-23 17:33:00 on dist-test-slave-p7jt"
server_key: "a56204c0471e9ffa0a1d571fa09e7deb"
server_key_iv: "415a44487c9c5770e52d322078f8b9fd"
server_key_version: "encryptionkey@0"
I20260923 17:33:00.087392 25978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:00.106654 25978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:00.107537 25978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:00.107810 25978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:00.125658 25978 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:38081
I20260923 17:33:00.125676 26043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:38081 every 8 connection(s)
I20260923 17:33:00.127089 25978 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:00.131038 26044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:00.133060 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 25978
I20260923 17:33:00.133247 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:00.133798 8311 external_mini_cluster.cc:1428] Setting key 8f482eea6d34b5d020377d358ab457c1
I20260923 17:33:00.143430 26044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf: Bootstrap starting.
I20260923 17:33:00.146979 26044 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:00.148020 26044 log.cc:824] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:00.148057 26047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:00.135800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.8.29.254:38081)
0923 17:33:00.136880 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:00.136894 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:00.136937 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:33:00.138721 (+ 1784us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:00.138745 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:00.139228 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:00.139569 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:00.140141 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:00.141531 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:00.143450 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:00.144242 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:00.144383 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:00.145655 (+ 1272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:00.145662 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:00.145685 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:00.146222 (+ 537us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:00.146314 (+ 92us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:00.146336 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:00.146421 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:00.146707 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 17:33:00.147036 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":612,"thread_start_us":385,"threads_started":1}
I20260923 17:33:00.151039 26044 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf: No bootstrap required, opened a new log
I20260923 17:33:00.158447 26044 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4701be8736f5445d824598098624bfaf" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38081 } }
I20260923 17:33:00.158917 26044 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:00.159009 26044 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4701be8736f5445d824598098624bfaf, State: Initialized, Role: FOLLOWER
I20260923 17:33:00.159555 26044 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4701be8736f5445d824598098624bfaf" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38081 } }
I20260923 17:33:00.159775 26044 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:00.159894 26044 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:00.160045 26044 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:00.162443 26044 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4701be8736f5445d824598098624bfaf" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38081 } }
I20260923 17:33:00.163075 26044 leader_election.cc:302] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4701be8736f5445d824598098624bfaf; no voters:
I20260923 17:33:00.163630 26044 leader_election.cc:288] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:00.163776 26049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:00.164453 26049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [term 1 LEADER]: Becoming Leader. State: Replica: 4701be8736f5445d824598098624bfaf, State: Running, Role: LEADER
I20260923 17:33:00.164970 26049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4701be8736f5445d824598098624bfaf" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38081 } }
I20260923 17:33:00.165681 26044 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:00.167527 26051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4701be8736f5445d824598098624bfaf. Latest consensus state: current_term: 1 leader_uuid: "4701be8736f5445d824598098624bfaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4701be8736f5445d824598098624bfaf" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38081 } } }
I20260923 17:33:00.167968 26051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:00.167816 26050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4701be8736f5445d824598098624bfaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4701be8736f5445d824598098624bfaf" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38081 } } }
I20260923 17:33:00.168071 26050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:00.168978 26057 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:00.175282 26057 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:00.184725 26057 catalog_manager.cc:1347] Generated new cluster ID: 223f75dda1874b5b9d5cfda336c9bc59
I20260923 17:33:00.184860 26057 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:00.212117 26057 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:00.213500 26057 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:00.220856 26057 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf: Generated new TSK 0
I20260923 17:33:00.221915 26057 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:00.236712 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38081
--builtin_ntp_servers=127.8.29.212:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:33:00.585973 26068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:00.586295 26068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:00.586391 26068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:00.594257 26068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:00.594393 26068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:00.594563 26068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:33:00.604485 26068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:00.605830 26068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:00.607772 26068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:00.621096 26073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:00.621074 26074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:00.621043 26076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:00.622229 26068 server_base.cc:1034] running on GCE node
I20260923 17:33:00.622865 26068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:00.623773 26068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:00.625013 26068 hybrid_clock.cc:648] HybridClock initialized: now 1790184780624957 us; error 37 us; skew 500 ppm
I20260923 17:33:00.625350 26068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:00.627767 26068 webserver.cc:460] Webserver started at http://127.8.29.193:34271/ using document root <none> and password file <none>
I20260923 17:33:00.628440 26068 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:00.628535 26068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:00.628839 26068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:00.631933 26068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "1b8e5e082f514895a72c2ea1d69d5478"
format_stamp: "Formatted at 2026-09-23 17:33:00 on dist-test-slave-p7jt"
server_key: "e0483bf45b5acb41ca84d5f1680620c8"
server_key_iv: "f535f3677e2e2ad6d94eeff87868a900"
server_key_version: "encryptionkey@0"
I20260923 17:33:00.632787 26068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "1b8e5e082f514895a72c2ea1d69d5478"
format_stamp: "Formatted at 2026-09-23 17:33:00 on dist-test-slave-p7jt"
server_key: "e0483bf45b5acb41ca84d5f1680620c8"
server_key_iv: "f535f3677e2e2ad6d94eeff87868a900"
server_key_version: "encryptionkey@0"
I20260923 17:33:00.638428 26068 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260923 17:33:00.642159 26082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:00.643553 26068 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260923 17:33:00.643702 26068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "1b8e5e082f514895a72c2ea1d69d5478"
format_stamp: "Formatted at 2026-09-23 17:33:00 on dist-test-slave-p7jt"
server_key: "e0483bf45b5acb41ca84d5f1680620c8"
server_key_iv: "f535f3677e2e2ad6d94eeff87868a900"
server_key_version: "encryptionkey@0"
I20260923 17:33:00.643940 26068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:00.661742 26068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:00.662681 26068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:00.662959 26068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:00.663931 26068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:00.665437 26068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:00.665527 26068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:00.665628 26068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:00.665688 26068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:00.695691 26068 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:46121
I20260923 17:33:00.695746 26194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:46121 every 8 connection(s)
I20260923 17:33:00.697022 26068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:33:00.698906 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 26068
I20260923 17:33:00.699085 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:33:00.699525 8311 external_mini_cluster.cc:1428] Setting key ca6211de7170e16be0aeffdb422c0ae2
I20260923 17:33:00.702422 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38081
--builtin_ntp_servers=127.8.29.212:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:33:00.710279 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:00.699933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:48945 (local address 127.8.29.254:38081)
0923 17:33:00.700413 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:00.700418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:00.701853 (+ 1435us) server_negotiation.cc:407] Connection header received
0923 17:33:00.703787 (+ 1934us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:00.703793 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:00.703875 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:00.703969 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:00.705112 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:00.705820 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:00.707459 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:00.707875 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:00.707984 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:00.709522 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:00.709527 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:00.709530 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:00.709549 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:00.709592 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:00.709604 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:00.709680 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:00.709924 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 17:33:00.710011 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":234,"threads_started":1}
I20260923 17:33:00.711670 26198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:00.700619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38081 (local address 127.8.29.193:48945)
0923 17:33:00.701712 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:00.701752 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:00.703508 (+ 1756us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:00.704241 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:00.704249 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:00.704353 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:00.704946 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:00.704964 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:00.705961 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:00.705965 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:00.707311 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:00.707325 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:00.708100 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:00.708108 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:00.708342 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:00.709102 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:33:00.709106 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:00.709173 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:33:00.709305 (+ 132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:00.709713 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:00.709724 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:00.709732 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 17:33:00.709913 (+ 181us) client_negotiation.cc:240] Negotiation successful
0923 17:33:00.710333 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":290,"threads_started":1}
I20260923 17:33:00.713353 26195 heartbeater.cc:347] Connected to a master server at 127.8.29.254:38081
I20260923 17:33:00.713660 26195 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:00.714612 26195 heartbeater.cc:511] Master 127.8.29.254:38081 requested a full tablet report, sending...
I20260923 17:33:00.717015 26009 ts_manager.cc:194] Registered new tserver with Master: 1b8e5e082f514895a72c2ea1d69d5478 (127.8.29.193:46121)
I20260923 17:33:00.719035 26009 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.193:48945
W20260923 17:33:01.031934 26199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:01.032228 26199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:01.032311 26199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:01.039714 26199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:01.039835 26199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:01.039944 26199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:33:01.049332 26199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:01.050668 26199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:01.052385 26199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:01.065688 26207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:01.065745 26204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:01.065745 26205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:01.066653 26199 server_base.cc:1034] running on GCE node
I20260923 17:33:01.067302 26199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:01.068244 26199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:01.069442 26199 hybrid_clock.cc:648] HybridClock initialized: now 1790184781069388 us; error 37 us; skew 500 ppm
I20260923 17:33:01.069799 26199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:01.072024 26199 webserver.cc:460] Webserver started at http://127.8.29.194:36843/ using document root <none> and password file <none>
I20260923 17:33:01.072739 26199 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:01.072840 26199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:01.073141 26199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:01.076417 26199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "35050cdcd86b4385b8e2ea145db2bc60"
format_stamp: "Formatted at 2026-09-23 17:33:01 on dist-test-slave-p7jt"
server_key: "f1c27a47dad3d5725d22cedbf452fb0f"
server_key_iv: "6faa34b3e847c024344523213ab5d2d6"
server_key_version: "encryptionkey@0"
I20260923 17:33:01.077322 26199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "35050cdcd86b4385b8e2ea145db2bc60"
format_stamp: "Formatted at 2026-09-23 17:33:01 on dist-test-slave-p7jt"
server_key: "f1c27a47dad3d5725d22cedbf452fb0f"
server_key_iv: "6faa34b3e847c024344523213ab5d2d6"
server_key_version: "encryptionkey@0"
I20260923 17:33:01.082902 26199 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260923 17:33:01.086585 26213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:01.087880 26199 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:01.088038 26199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "35050cdcd86b4385b8e2ea145db2bc60"
format_stamp: "Formatted at 2026-09-23 17:33:01 on dist-test-slave-p7jt"
server_key: "f1c27a47dad3d5725d22cedbf452fb0f"
server_key_iv: "6faa34b3e847c024344523213ab5d2d6"
server_key_version: "encryptionkey@0"
I20260923 17:33:01.088255 26199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:01.111475 26199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:01.112495 26199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:01.112875 26199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:01.113879 26199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:01.115456 26199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:01.115563 26199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:01.115691 26199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:01.115768 26199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:01.148612 26199 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:42267
I20260923 17:33:01.148658 26325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:42267 every 8 connection(s)
I20260923 17:33:01.149991 26199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:33:01.159309 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 26199
I20260923 17:33:01.159467 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:33:01.160029 8311 external_mini_cluster.cc:1428] Setting key dbe8506df0f9ff587708e4f1de78d125
I20260923 17:33:01.162339 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.152550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:56459 (local address 127.8.29.254:38081)
0923 17:33:01.152751 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:01.152755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:01.154071 (+ 1316us) server_negotiation.cc:407] Connection header received
0923 17:33:01.155935 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:01.155939 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:01.156009 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:01.156095 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:01.157160 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.157878 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.159466 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.160053 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.160188 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.161443 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:01.161449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:01.161453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:01.161472 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:01.161509 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:01.161524 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:01.161614 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:01.161913 (+ 299us) server_negotiation.cc:299] Negotiation successful
0923 17:33:01.162015 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:33:01.162881 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38081
--builtin_ntp_servers=127.8.29.212:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:33:01.163480 26328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.152982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38081 (local address 127.8.29.194:56459)
0923 17:33:01.153908 (+ 926us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:01.153949 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:01.155697 (+ 1748us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:01.156341 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:01.156350 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:01.156447 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:01.157022 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.157039 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.158014 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.158018 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.159311 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.159323 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.160283 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.160289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.160491 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.161135 (+ 644us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:33:01.161137 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:01.161181 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:33:01.161269 (+ 88us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:01.161653 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:01.161665 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:01.161674 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:33:01.161876 (+ 202us) client_negotiation.cc:240] Negotiation successful
0923 17:33:01.162310 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":267,"threads_started":1}
I20260923 17:33:01.165035 26326 heartbeater.cc:347] Connected to a master server at 127.8.29.254:38081
I20260923 17:33:01.165354 26326 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:01.166155 26326 heartbeater.cc:511] Master 127.8.29.254:38081 requested a full tablet report, sending...
I20260923 17:33:01.167645 26009 ts_manager.cc:194] Registered new tserver with Master: 35050cdcd86b4385b8e2ea145db2bc60 (127.8.29.194:42267)
I20260923 17:33:01.168447 26009 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.194:56459
W20260923 17:33:01.486946 26329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:01.487231 26329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:01.487313 26329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:01.494801 26329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:01.494915 26329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:01.495030 26329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:33:01.504335 26329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:38081
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:01.505673 26329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:01.507431 26329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:01.520417 26334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:01.520449 26337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:01.520481 26335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:01.521298 26329 server_base.cc:1034] running on GCE node
I20260923 17:33:01.521915 26329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:01.522929 26329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:01.524195 26329 hybrid_clock.cc:648] HybridClock initialized: now 1790184781524111 us; error 61 us; skew 500 ppm
I20260923 17:33:01.524545 26329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:01.527156 26329 webserver.cc:460] Webserver started at http://127.8.29.195:46001/ using document root <none> and password file <none>
I20260923 17:33:01.527930 26329 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:01.528038 26329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:01.528362 26329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:01.531435 26329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "6755f87af3ee4722bd75478455af9d36"
format_stamp: "Formatted at 2026-09-23 17:33:01 on dist-test-slave-p7jt"
server_key: "502f81df56e56b4945ff7f62c2c4c391"
server_key_iv: "39fc0ea496eefbfc29d247e8519e6fcc"
server_key_version: "encryptionkey@0"
I20260923 17:33:01.532249 26329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "6755f87af3ee4722bd75478455af9d36"
format_stamp: "Formatted at 2026-09-23 17:33:01 on dist-test-slave-p7jt"
server_key: "502f81df56e56b4945ff7f62c2c4c391"
server_key_iv: "39fc0ea496eefbfc29d247e8519e6fcc"
server_key_version: "encryptionkey@0"
I20260923 17:33:01.537988 26329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:33:01.541778 26343 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:01.543219 26329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:01.543380 26329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "6755f87af3ee4722bd75478455af9d36"
format_stamp: "Formatted at 2026-09-23 17:33:01 on dist-test-slave-p7jt"
server_key: "502f81df56e56b4945ff7f62c2c4c391"
server_key_iv: "39fc0ea496eefbfc29d247e8519e6fcc"
server_key_version: "encryptionkey@0"
I20260923 17:33:01.543591 26329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:01.561416 26329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:01.562288 26329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:01.562620 26329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:01.563643 26329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:01.565276 26329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:01.565382 26329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:01.565487 26329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:01.565554 26329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:01.596314 26329 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:38433
I20260923 17:33:01.596371 26455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:38433 every 8 connection(s)
I20260923 17:33:01.597635 26329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:33:01.598074 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 26329
I20260923 17:33:01.598240 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:33:01.598901 8311 external_mini_cluster.cc:1428] Setting key 7a05abf57ccf41636fd55548e8eee9bb
I20260923 17:33:01.610971 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.600295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:32855 (local address 127.8.29.254:38081)
0923 17:33:01.600477 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:01.600482 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:01.602541 (+ 2059us) server_negotiation.cc:407] Connection header received
0923 17:33:01.604346 (+ 1805us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:01.604350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:01.604419 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:01.604513 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:01.605689 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.606536 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.608196 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.608641 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.608774 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.610131 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:01.610135 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:01.610139 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:01.610159 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:01.610192 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:01.610204 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:01.610289 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:01.610599 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 17:33:01.610691 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:33:01.612044 26458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.601121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38081 (local address 127.8.29.195:32855)
0923 17:33:01.602368 (+ 1247us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:01.602420 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:01.604075 (+ 1655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:01.604818 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:01.604827 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:01.604924 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:01.605517 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.605535 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.606688 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.606693 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.608027 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.608041 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.608881 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.608888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.609111 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.609793 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:33:01.609796 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:01.609839 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:33:01.609924 (+ 85us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:01.610339 (+ 415us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:01.610351 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:01.610358 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:33:01.610572 (+ 214us) client_negotiation.cc:240] Negotiation successful
0923 17:33:01.610915 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":328,"threads_started":1}
I20260923 17:33:01.613706 26456 heartbeater.cc:347] Connected to a master server at 127.8.29.254:38081
I20260923 17:33:01.614027 26456 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:01.614936 26456 heartbeater.cc:511] Master 127.8.29.254:38081 requested a full tablet report, sending...
I20260923 17:33:01.616454 26008 ts_manager.cc:194] Registered new tserver with Master: 6755f87af3ee4722bd75478455af9d36 (127.8.29.195:38433)
I20260923 17:33:01.617658 26008 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.195:32855
I20260923 17:33:01.627308 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:33:01.637382 26197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.631449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50244 (local address 127.8.29.254:38081)
0923 17:33:01.631759 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:01.631767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:01.632002 (+ 235us) server_negotiation.cc:407] Connection header received
0923 17:33:01.632184 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:01.632188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:01.632272 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:01.632362 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:01.632901 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.633601 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.634961 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.635389 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.635503 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.636594 (+ 1091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:01.636600 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:01.636604 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:01.636624 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:01.636664 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:01.636681 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:01.636778 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:01.636934 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 17:33:01.637053 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:33:01.641571 26008 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:33:01.648720 26008 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:33:01.682941 26469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.660990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:38433 (local address 127.0.0.1:60822)
0923 17:33:01.673682 (+ 12692us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:01.673762 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:01.674107 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:01.675158 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:01.675166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:01.675215 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:01.675698 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.675712 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.677928 (+ 2216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.677933 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.680198 (+ 2265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.680212 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.681893 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.681899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.682097 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.682156 (+ 59us) client_negotiation.cc:769] Sending connection context
0923 17:33:01.682307 (+ 151us) client_negotiation.cc:240] Negotiation successful
0923 17:33:01.682536 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12378,"mutex_wait_us":58,"spinlock_wait_cycles":6656,"thread_start_us":12819,"threads_started":1}
I20260923 17:33:01.683254 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.661582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43498 (local address 127.8.29.194:42267)
0923 17:33:01.673622 (+ 12040us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:01.673630 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:01.674196 (+ 566us) server_negotiation.cc:407] Connection header received
0923 17:33:01.674502 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:01.674509 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:01.674696 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:01.674893 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:01.676894 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.678213 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.680788 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.682150 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.682278 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.682543 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:01.682755 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 17:33:01.682935 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11863,"thread_start_us":11746,"threads_started":1}
I20260923 17:33:01.683889 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.661724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60822 (local address 127.8.29.195:38433)
0923 17:33:01.673160 (+ 11436us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:01.673172 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:01.673910 (+ 738us) server_negotiation.cc:407] Connection header received
0923 17:33:01.674358 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:01.674367 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:01.674660 (+ 293us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:01.674848 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:01.676360 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.677799 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.680390 (+ 2591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.681698 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.681835 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.682125 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:01.683190 (+ 1065us) server_negotiation.cc:299] Negotiation successful
0923 17:33:01.683549 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11225,"thread_start_us":10893,"threads_started":1}
I20260923 17:33:01.683616 26472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.674737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46121 (local address 127.0.0.1:52632)
0923 17:33:01.675882 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:01.675898 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:01.676012 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:01.677079 (+ 1067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:01.677083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:01.677111 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:01.677439 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.677452 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.679063 (+ 1611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.679069 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.681084 (+ 2015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.681095 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.682902 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.682908 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.683071 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.683106 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 17:33:01.683190 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:33:01.683316 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1018,"thread_start_us":865,"threads_started":1}
I20260923 17:33:01.684252 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.674805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52632 (local address 127.8.29.193:46121)
0923 17:33:01.676498 (+ 1693us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:01.676507 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:01.676537 (+ 30us) server_negotiation.cc:407] Connection header received
0923 17:33:01.676630 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:01.676637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:01.676892 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:01.677095 (+ 203us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:01.677593 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.678898 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.681254 (+ 2356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.682392 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.682532 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.682767 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:01.683765 (+ 998us) server_negotiation.cc:299] Negotiation successful
0923 17:33:01.683934 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1530,"thread_start_us":1406,"threads_started":1}
I20260923 17:33:01.683974 26468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.661200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:42267 (local address 127.0.0.1:43498)
0923 17:33:01.674084 (+ 12884us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:01.674102 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:01.674310 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:01.675158 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:01.675166 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:01.675207 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:01.676144 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.676157 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.678365 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.678370 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.680563 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.680583 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.682331 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.682337 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.682538 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.682571 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:33:01.683455 (+ 884us) client_negotiation.cc:240] Negotiation successful
0923 17:33:01.683588 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12767,"mutex_wait_us":395,"spinlock_wait_cycles":1280,"thread_start_us":12371,"threads_started":1}
I20260923 17:33:01.687877 26261 tablet_service.cc:1467] Processing CreateTablet for tablet a7576b5f224c436a9e16560923098763 (DEFAULT_TABLE table=test-table [id=7f30957e7ca94879a44e26b3e1f77549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:01.687976 26391 tablet_service.cc:1467] Processing CreateTablet for tablet a7576b5f224c436a9e16560923098763 (DEFAULT_TABLE table=test-table [id=7f30957e7ca94879a44e26b3e1f77549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:01.688467 26130 tablet_service.cc:1467] Processing CreateTablet for tablet a7576b5f224c436a9e16560923098763 (DEFAULT_TABLE table=test-table [id=7f30957e7ca94879a44e26b3e1f77549]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:01.689265 26261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7576b5f224c436a9e16560923098763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:01.689414 26391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7576b5f224c436a9e16560923098763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:01.689641 26130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a7576b5f224c436a9e16560923098763. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:01.709352 26474 tablet_bootstrap.cc:492] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: Bootstrap starting.
I20260923 17:33:01.709844 26475 tablet_bootstrap.cc:492] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478: Bootstrap starting.
I20260923 17:33:01.713042 26474 tablet_bootstrap.cc:654] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:01.713824 26475 tablet_bootstrap.cc:654] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:01.715083 26474 log.cc:824] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:01.715994 26476 tablet_bootstrap.cc:492] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: Bootstrap starting.
I20260923 17:33:01.717042 26475 log.cc:824] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:01.718573 26474 tablet_bootstrap.cc:492] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: No bootstrap required, opened a new log
I20260923 17:33:01.718870 26474 ts_tablet_manager.cc:1397] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260923 17:33:01.719642 26475 tablet_bootstrap.cc:492] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478: No bootstrap required, opened a new log
I20260923 17:33:01.719879 26475 ts_tablet_manager.cc:1397] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 17:33:01.720041 26476 tablet_bootstrap.cc:654] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:01.721243 26476 log.cc:824] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:01.722447 26195 heartbeater.cc:503] Master 127.8.29.254:38081 was elected leader, sending a full tablet report...
I20260923 17:33:01.726424 26476 tablet_bootstrap.cc:492] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: No bootstrap required, opened a new log
I20260923 17:33:01.726815 26476 ts_tablet_manager.cc:1397] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.002s
I20260923 17:33:01.728701 26474 raft_consensus.cc:348] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } }
I20260923 17:33:01.729086 26474 raft_consensus.cc:374] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:01.729212 26474 raft_consensus.cc:729] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35050cdcd86b4385b8e2ea145db2bc60, State: Initialized, Role: FOLLOWER
I20260923 17:33:01.729523 26475 raft_consensus.cc:348] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } }
I20260923 17:33:01.729974 26475 raft_consensus.cc:374] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:01.730072 26475 raft_consensus.cc:729] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b8e5e082f514895a72c2ea1d69d5478, State: Initialized, Role: FOLLOWER
I20260923 17:33:01.729912 26474 consensus_queue.cc:260] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } }
I20260923 17:33:01.730688 26475 consensus_queue.cc:260] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } }
I20260923 17:33:01.732525 26475 ts_tablet_manager.cc:1428] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260923 17:33:01.733451 26326 heartbeater.cc:503] Master 127.8.29.254:38081 was elected leader, sending a full tablet report...
I20260923 17:33:01.734161 26474 ts_tablet_manager.cc:1428] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
I20260923 17:33:01.737180 26476 raft_consensus.cc:348] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } }
I20260923 17:33:01.737602 26476 raft_consensus.cc:374] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:01.737699 26476 raft_consensus.cc:729] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6755f87af3ee4722bd75478455af9d36, State: Initialized, Role: FOLLOWER
I20260923 17:33:01.738329 26476 consensus_queue.cc:260] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } }
I20260923 17:33:01.739434 26456 heartbeater.cc:503] Master 127.8.29.254:38081 was elected leader, sending a full tablet report...
I20260923 17:33:01.739880 26476 ts_tablet_manager.cc:1428] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: Time spent starting tablet: real 0.013s user 0.010s sys 0.002s
W20260923 17:33:01.849450 26457 tablet.cc:2369] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:01.897620 26480 raft_consensus.cc:482] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:33:01.897980 26480 raft_consensus.cc:504] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } }
I20260923 17:33:01.899725 26480 leader_election.cc:288] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6755f87af3ee4722bd75478455af9d36 (127.8.29.195:38433), 1b8e5e082f514895a72c2ea1d69d5478 (127.8.29.193:46121)
W20260923 17:33:01.902033 26327 tablet.cc:2369] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:01.907241 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.900204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:58137 (local address 127.8.29.195:38433)
0923 17:33:01.900404 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:01.900409 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:01.900918 (+ 509us) server_negotiation.cc:407] Connection header received
0923 17:33:01.901102 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:01.901108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:01.901190 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:01.901309 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:01.902027 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.902958 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.905285 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.906401 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.906599 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.906672 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:01.906907 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 17:33:01.907038 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:33:01.907300 26483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.900091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:38433 (local address 127.8.29.194:58137)
0923 17:33:01.900784 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:01.900803 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:01.900950 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:01.901425 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:01.901430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:01.901466 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:01.901868 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.901880 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.903074 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.903079 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.905070 (+ 1991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.905086 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.906600 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.906604 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.906724 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.906780 (+ 56us) client_negotiation.cc:769] Sending connection context
0923 17:33:01.906853 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 17:33:01.906970 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":362,"threads_started":1}
I20260923 17:33:01.907739 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.901245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:60245 (local address 127.8.29.193:46121)
0923 17:33:01.901443 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:01.901449 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:01.901591 (+ 142us) server_negotiation.cc:407] Connection header received
0923 17:33:01.901764 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:01.901769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:01.901846 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:01.901951 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:01.902579 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.903426 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.905618 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.906855 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.907009 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.907069 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:01.907402 (+ 333us) server_negotiation.cc:299] Negotiation successful
0923 17:33:01.907537 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:33:01.908218 26484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.900918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46121 (local address 127.8.29.194:60245)
0923 17:33:01.901470 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:01.901485 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:01.901623 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:01.901985 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:01.901989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:01.902020 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:01.902391 (+ 371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.902403 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.903568 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.903573 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.905451 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:01.905461 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.907032 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.907038 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:01.907200 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.907229 (+ 29us) client_negotiation.cc:769] Sending connection context
0923 17:33:01.907905 (+ 676us) client_negotiation.cc:240] Negotiation successful
0923 17:33:01.908024 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":272,"threads_started":1}
I20260923 17:33:01.908234 26411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7576b5f224c436a9e16560923098763" candidate_uuid: "35050cdcd86b4385b8e2ea145db2bc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6755f87af3ee4722bd75478455af9d36" is_pre_election: true
I20260923 17:33:01.908773 26411 raft_consensus.cc:2454] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35050cdcd86b4385b8e2ea145db2bc60 in term 0.
I20260923 17:33:01.909135 26150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7576b5f224c436a9e16560923098763" candidate_uuid: "35050cdcd86b4385b8e2ea145db2bc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b8e5e082f514895a72c2ea1d69d5478" is_pre_election: true
I20260923 17:33:01.909435 26216 leader_election.cc:302] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35050cdcd86b4385b8e2ea145db2bc60, 6755f87af3ee4722bd75478455af9d36; no voters:
I20260923 17:33:01.909678 26150 raft_consensus.cc:2454] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35050cdcd86b4385b8e2ea145db2bc60 in term 0.
I20260923 17:33:01.909873 26480 raft_consensus.cc:2781] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:33:01.910003 26480 raft_consensus.cc:482] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:33:01.910121 26480 raft_consensus.cc:3037] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:01.912346 26480 raft_consensus.cc:504] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } }
I20260923 17:33:01.912912 26480 leader_election.cc:288] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [CANDIDATE]: Term 1 election: Requested vote from peers 6755f87af3ee4722bd75478455af9d36 (127.8.29.195:38433), 1b8e5e082f514895a72c2ea1d69d5478 (127.8.29.193:46121)
I20260923 17:33:01.913415 26411 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7576b5f224c436a9e16560923098763" candidate_uuid: "35050cdcd86b4385b8e2ea145db2bc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6755f87af3ee4722bd75478455af9d36"
I20260923 17:33:01.913503 26150 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a7576b5f224c436a9e16560923098763" candidate_uuid: "35050cdcd86b4385b8e2ea145db2bc60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b8e5e082f514895a72c2ea1d69d5478"
I20260923 17:33:01.913599 26411 raft_consensus.cc:3037] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:01.913714 26150 raft_consensus.cc:3037] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:01.915877 26411 raft_consensus.cc:2454] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35050cdcd86b4385b8e2ea145db2bc60 in term 1.
I20260923 17:33:01.916148 26150 raft_consensus.cc:2454] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35050cdcd86b4385b8e2ea145db2bc60 in term 1.
I20260923 17:33:01.916375 26216 leader_election.cc:302] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35050cdcd86b4385b8e2ea145db2bc60, 6755f87af3ee4722bd75478455af9d36; no voters:
I20260923 17:33:01.916754 26480 raft_consensus.cc:2781] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:01.917244 26480 raft_consensus.cc:686] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 1 LEADER]: Becoming Leader. State: Replica: 35050cdcd86b4385b8e2ea145db2bc60, State: Running, Role: LEADER
I20260923 17:33:01.917791 26480 consensus_queue.cc:237] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } }
I20260923 17:33:01.922262 26008 catalog_manager.cc:5495] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 reported cstate change: term changed from 0 to 1, leader changed from <none> to 35050cdcd86b4385b8e2ea145db2bc60 (127.8.29.194). New cstate: current_term: 1 leader_uuid: "35050cdcd86b4385b8e2ea145db2bc60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35050cdcd86b4385b8e2ea145db2bc60" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 42267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 } health_report { overall_health: UNKNOWN } } }
W20260923 17:33:01.949360 26196 tablet.cc:2369] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:01.991808 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:01.986315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43504 (local address 127.8.29.194:42267)
0923 17:33:01.986520 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:01.986524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:01.986599 (+ 75us) server_negotiation.cc:407] Connection header received
0923 17:33:01.986780 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:01.986784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:01.986900 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:01.987036 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:01.987472 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.988241 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.989618 (+ 1377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:01.990052 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:01.990150 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:01.990743 (+ 593us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:01.991272 (+ 529us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:01.991411 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:01.991458 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 17:33:01.991551 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:33:02.000355 26150 raft_consensus.cc:1261] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [term 1 FOLLOWER]: Refusing update from remote peer 35050cdcd86b4385b8e2ea145db2bc60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:02.000401 26411 raft_consensus.cc:1261] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [term 1 FOLLOWER]: Refusing update from remote peer 35050cdcd86b4385b8e2ea145db2bc60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:02.001294 26485 consensus_queue.cc:1035] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b8e5e082f514895a72c2ea1d69d5478" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:02.001662 26480 consensus_queue.cc:1035] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6755f87af3ee4722bd75478455af9d36" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 38433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:02.013002 26488 mvcc.cc:205] Tried to move back new op lower bound from 7332596867063005184 to 7332596866739224576. Current Snapshot: MvccSnapshot[applied={T|T < 7332596867063005184}]
I20260923 17:33:02.018622 26489 mvcc.cc:205] Tried to move back new op lower bound from 7332596867063005184 to 7332596866739224576. Current Snapshot: MvccSnapshot[applied={T|T < 7332596867063005184 or (T in {7332596867063005184})}]
I20260923 17:33:02.026260 26009 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:02.026571 26009 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:02.030634 26009 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4701be8736f5445d824598098624bfaf: Sending DeleteTablet for 3 replicas of tablet a7576b5f224c436a9e16560923098763
I20260923 17:33:02.031908 26261 tablet_service.cc:1514] Processing DeleteTablet for tablet a7576b5f224c436a9e16560923098763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:02 UTC) from {username='slave'} at 127.0.0.1:43498
I20260923 17:33:02.031968 26391 tablet_service.cc:1514] Processing DeleteTablet for tablet a7576b5f224c436a9e16560923098763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:02 UTC) from {username='slave'} at 127.0.0.1:60822
I20260923 17:33:02.032040 26130 tablet_service.cc:1514] Processing DeleteTablet for tablet a7576b5f224c436a9e16560923098763 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:02 UTC) from {username='slave'} at 127.0.0.1:52632
I20260923 17:33:02.034011 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 26068
I20260923 17:33:02.047395 26499 tablet_replica.cc:330] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: stopping tablet replica
I20260923 17:33:02.047689 26498 tablet_replica.cc:330] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: stopping tablet replica
I20260923 17:33:02.048046 26499 raft_consensus.cc:2229] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:02.048220 26498 raft_consensus.cc:2229] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:33:02.048620 26499 raft_consensus.cc:2258] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:02.049155 26498 raft_consensus.cc:2258] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:02.051402 26499 ts_tablet_manager.cc:1905] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:02.051410 26498 ts_tablet_manager.cc:1905] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:02.054141 26500 tablet_replica.cc:330] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478: stopping tablet replica
I20260923 17:33:02.054616 26500 raft_consensus.cc:2229] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:02.055121 26500 raft_consensus.cc:2258] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:02.058768 26500 ts_tablet_manager.cc:1905] T a7576b5f224c436a9e16560923098763 P 1b8e5e082f514895a72c2ea1d69d5478: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:02.059056 26499 ts_tablet_manager.cc:1918] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:02.059129 26498 ts_tablet_manager.cc:1918] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:02.059324 26499 log.cc:1196] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/a7576b5f224c436a9e16560923098763
I20260923 17:33:02.059319 26498 log.cc:1196] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/a7576b5f224c436a9e16560923098763
I20260923 17:33:02.059870 26498 ts_tablet_manager.cc:1939] T a7576b5f224c436a9e16560923098763 P 35050cdcd86b4385b8e2ea145db2bc60: Deleting consensus metadata
I20260923 17:33:02.059893 26499 ts_tablet_manager.cc:1939] T a7576b5f224c436a9e16560923098763 P 6755f87af3ee4722bd75478455af9d36: Deleting consensus metadata
I20260923 17:33:02.061761 25994 catalog_manager.cc:4841] TS 35050cdcd86b4385b8e2ea145db2bc60 (127.8.29.194:42267): tablet a7576b5f224c436a9e16560923098763 (table test-table [id=7f30957e7ca94879a44e26b3e1f77549]) successfully deleted
I20260923 17:33:02.062784 26473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.035165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52638 (local address 127.8.29.193:46121)
0923 17:33:02.035409 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:02.035416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:02.049442 (+ 14026us) server_negotiation.cc:407] Connection header received
0923 17:33:02.049853 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:02.049858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:02.049991 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:02.050083 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:02.050766 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:02.051847 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:02.055529 (+ 3682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:02.056145 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:02.056263 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:02.061058 (+ 4795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:02.061071 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:02.061090 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:02.061856 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:02.061926 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:02.061953 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:02.062052 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:02.062294 (+ 242us) server_negotiation.cc:299] Negotiation successful
0923 17:33:02.062415 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260923 17:33:02.063043 25995 catalog_manager.cc:4841] TS 6755f87af3ee4722bd75478455af9d36 (127.8.29.195:38433): tablet a7576b5f224c436a9e16560923098763 (table test-table [id=7f30957e7ca94879a44e26b3e1f77549]) successfully deleted
I20260923 17:33:02.063933 26190 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:02.178195 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 26068
I20260923 17:33:02.201151 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 26199
W20260923 17:33:02.201287 25995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.29.193:46121 (error 108)
I20260923 17:33:02.203953 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.203219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46121 (local address 127.0.0.1:52648)
0923 17:33:02.203665 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:02.203774 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":231,"threads_started":1}
W20260923 17:33:02.204288 25995 catalog_manager.cc:4584] TS 1b8e5e082f514895a72c2ea1d69d5478 (127.8.29.193:46121): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a7576b5f224c436a9e16560923098763: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46121: connect: Connection refused (error 111)
I20260923 17:33:02.210083 26470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.202383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43508 (local address 127.8.29.194:42267)
0923 17:33:02.202586 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:02.202590 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:02.202606 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:33:02.202658 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:02.202662 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:02.202744 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:02.202838 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:02.203357 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:02.204564 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:02.205924 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:02.206895 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:02.207044 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:02.208306 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:02.208319 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:02.208339 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:02.209190 (+ 851us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:02.209256 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:02.209282 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:02.209391 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:02.209559 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 17:33:02.209674 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260923 17:33:02.210875 26321 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:02.257941 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.257415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46121 (local address 127.0.0.1:52652)
0923 17:33:02.257654 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:02.257763 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260923 17:33:02.259009 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.258649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46121 (local address 127.0.0.1:52662)
0923 17:33:02.258768 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:02.258840 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 17:33:02.319746 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.319250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46121 (local address 127.0.0.1:52672)
0923 17:33:02.319491 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:02.319589 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260923 17:33:02.320772 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.320423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46121 (local address 127.0.0.1:52678)
0923 17:33:02.320561 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:02.320637 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260923 17:33:02.325248 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 26199
I20260923 17:33:02.340852 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 26329
I20260923 17:33:02.347863 26471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.341581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60834 (local address 127.8.29.195:38433)
0923 17:33:02.341768 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:02.341773 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:02.341832 (+ 59us) server_negotiation.cc:407] Connection header received
0923 17:33:02.341996 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:02.342000 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:02.342085 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:02.342176 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:02.342710 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:02.343437 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:02.344754 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:02.345162 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:02.345290 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:02.346311 (+ 1021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:02.346321 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:02.346336 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:02.347111 (+ 775us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:02.347164 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:02.347186 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:02.347285 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:02.347428 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 17:33:02.347525 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:33:02.348513 26451 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:02.401590 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.400970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46121 (local address 127.0.0.1:52684)
0923 17:33:02.401288 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:02.401405 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260923 17:33:02.402808 26504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:02.402296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46121 (local address 127.0.0.1:52700)
0923 17:33:02.402549 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:02.402640 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260923 17:33:02.437409 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 26329
I20260923 17:33:02.450860 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 25978
I20260923 17:33:02.452008 26039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:02.562873 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 25978
2026-09-23T17:33:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2935 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:02 dist-test-slave-p7jt krb5kdc[26510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:02 dist-test-slave-p7jt krb5kdc[26510](info): set up 2 sockets
Sep 23 17:33:02 dist-test-slave-p7jt krb5kdc[26510](info): commencing operation
krb5kdc: starting...
W20260923 17:33:03.602418 8311 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:03 dist-test-slave-p7jt krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:03.853616 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:36689
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:36689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:04.183465 26523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:04.183797 26523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:04.183936 26523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:04.191362 26523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:04.191494 26523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:04.191576 26523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:04.191650 26523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:04.191711 26523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:04.201985 26523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:36689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:36689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:04.203373 26523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:04.204692 26523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:04.216408 26528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:04.216339 26531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:04.216375 26529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:04.217732 26523 server_base.cc:1034] running on GCE node
I20260923 17:33:04.218662 26523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:04.220186 26523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:04.221415 26523 hybrid_clock.cc:648] HybridClock initialized: now 1790184784221366 us; error 40 us; skew 500 ppm
I20260923 17:33:04.221779 26523 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:04 dist-test-slave-p7jt krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184784, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:04.224967 26523 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:04.226691 26523 webserver.cc:460] Webserver started at http://127.8.29.254:35009/ using document root <none> and password file <none>
I20260923 17:33:04.227403 26523 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:04.227521 26523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:04.227823 26523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:04.230935 26523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "1f64d9ddb8db4825ac3dcfc52afb6732"
format_stamp: "Formatted at 2026-09-23 17:33:04 on dist-test-slave-p7jt"
server_key: "c3b440b59540f4301a3ef5b3bcfc884b"
server_key_iv: "c6785f7146d9ec723e0f269847f25d3b"
server_key_version: "encryptionkey@0"
I20260923 17:33:04.231736 26523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "1f64d9ddb8db4825ac3dcfc52afb6732"
format_stamp: "Formatted at 2026-09-23 17:33:04 on dist-test-slave-p7jt"
server_key: "c3b440b59540f4301a3ef5b3bcfc884b"
server_key_iv: "c6785f7146d9ec723e0f269847f25d3b"
server_key_version: "encryptionkey@0"
I20260923 17:33:04.237926 26523 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260923 17:33:04.241786 26538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:04.243306 26523 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:04.243505 26523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "1f64d9ddb8db4825ac3dcfc52afb6732"
format_stamp: "Formatted at 2026-09-23 17:33:04 on dist-test-slave-p7jt"
server_key: "c3b440b59540f4301a3ef5b3bcfc884b"
server_key_iv: "c6785f7146d9ec723e0f269847f25d3b"
server_key_version: "encryptionkey@0"
I20260923 17:33:04.243707 26523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:04.286429 26523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:04.291028 26523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:04.291316 26523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:04.309160 26523 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:36689
I20260923 17:33:04.309177 26589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:36689 every 8 connection(s)
I20260923 17:33:04.310559 26523 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:04.314330 26590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:04.317976 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 26523
I20260923 17:33:04.318164 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:04.318769 8311 external_mini_cluster.cc:1428] Setting key e99e6a9fbf6ade1a3014df9996d6a261
I20260923 17:33:04.326823 26590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732: Bootstrap starting.
I20260923 17:33:04.329931 26590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:04.331012 26590 log.cc:824] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732: Log is configured to *not* fsync() on all Append() calls
Sep 23 17:33:04 dist-test-slave-p7jt krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:04.334022 26590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732: No bootstrap required, opened a new log
I20260923 17:33:04.341069 26593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:04.320855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37910 (local address 127.8.29.254:36689)
0923 17:33:04.321648 (+ 793us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:04.321661 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:04.321736 (+ 75us) server_negotiation.cc:407] Connection header received
0923 17:33:04.323486 (+ 1750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:04.323510 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:04.323947 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:04.324319 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:04.325767 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:04.327166 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:04.328604 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:04.329308 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:04.329483 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:04.333548 (+ 4065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:04.333557 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:04.333573 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:04.333610 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:04.336681 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:04.337416 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:04.337426 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:04.337433 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:04.337620 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:04.338001 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:04.338004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:04.338006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:04.338662 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:04.338954 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:04.339468 (+ 514us) server_negotiation.cc:299] Negotiation successful
0923 17:33:04.339947 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":360,"threads_started":1}
I20260923 17:33:04.342213 26590 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f64d9ddb8db4825ac3dcfc52afb6732" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36689 } }
I20260923 17:33:04.342645 26590 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:04.342746 26590 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f64d9ddb8db4825ac3dcfc52afb6732, State: Initialized, Role: FOLLOWER
I20260923 17:33:04.343331 26590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f64d9ddb8db4825ac3dcfc52afb6732" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36689 } }
I20260923 17:33:04.343578 26590 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:04.343709 26590 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:04.343879 26590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:04.346289 26590 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f64d9ddb8db4825ac3dcfc52afb6732" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36689 } }
I20260923 17:33:04.346863 26590 leader_election.cc:302] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f64d9ddb8db4825ac3dcfc52afb6732; no voters:
I20260923 17:33:04.347508 26590 leader_election.cc:288] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:04.347661 26595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:04.349362 26595 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [term 1 LEADER]: Becoming Leader. State: Replica: 1f64d9ddb8db4825ac3dcfc52afb6732, State: Running, Role: LEADER
I20260923 17:33:04.349814 26590 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:04.349987 26595 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f64d9ddb8db4825ac3dcfc52afb6732" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36689 } }
I20260923 17:33:04.352962 26596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f64d9ddb8db4825ac3dcfc52afb6732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f64d9ddb8db4825ac3dcfc52afb6732" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36689 } } }
I20260923 17:33:04.352921 26597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f64d9ddb8db4825ac3dcfc52afb6732. Latest consensus state: current_term: 1 leader_uuid: "1f64d9ddb8db4825ac3dcfc52afb6732" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f64d9ddb8db4825ac3dcfc52afb6732" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36689 } } }
I20260923 17:33:04.353410 26596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:04.353410 26597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:04.354307 26605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:04.359758 26605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:04.368798 26605 catalog_manager.cc:1347] Generated new cluster ID: df94bc47093b4bd4b4a319258e65e919
I20260923 17:33:04.368929 26605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:04.386145 26605 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:04.388114 26605 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:04.400170 26605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732: Generated new TSK 0
I20260923 17:33:04.401366 26605 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:33:04.462997 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36689
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:04.800300 26616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:04.800587 26616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:04.800751 26616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:04.808142 26616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:04.808266 26616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:04.808411 26616 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:33:04.818444 26616 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:04.819778 26616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:04.821143 26616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:04.834959 26622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:04.834993 26624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:04.834950 26621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:04.835868 26616 server_base.cc:1034] running on GCE node
I20260923 17:33:04.836539 26616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:04.837505 26616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:04.838713 26616 hybrid_clock.cc:648] HybridClock initialized: now 1790184784838651 us; error 48 us; skew 500 ppm
I20260923 17:33:04.839056 26616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:04 dist-test-slave-p7jt krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184784, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:04.842316 26616 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:33:04.844095 26616 webserver.cc:460] Webserver started at http://127.8.29.193:39133/ using document root <none> and password file <none>
I20260923 17:33:04.844843 26616 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:04.844969 26616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:04.845278 26616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:04.848434 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "db0a54a671e34b16919a5935e708157b"
format_stamp: "Formatted at 2026-09-23 17:33:04 on dist-test-slave-p7jt"
server_key: "19e612fb1ca7f1604ab4ade9cb7bc9c3"
server_key_iv: "337ceb3345367ad855d4aea6dfc81c87"
server_key_version: "encryptionkey@0"
I20260923 17:33:04.849256 26616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "db0a54a671e34b16919a5935e708157b"
format_stamp: "Formatted at 2026-09-23 17:33:04 on dist-test-slave-p7jt"
server_key: "19e612fb1ca7f1604ab4ade9cb7bc9c3"
server_key_iv: "337ceb3345367ad855d4aea6dfc81c87"
server_key_version: "encryptionkey@0"
I20260923 17:33:04.855085 26616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:33:04.858896 26631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:04.860388 26616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:33:04.860554 26616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "db0a54a671e34b16919a5935e708157b"
format_stamp: "Formatted at 2026-09-23 17:33:04 on dist-test-slave-p7jt"
server_key: "19e612fb1ca7f1604ab4ade9cb7bc9c3"
server_key_iv: "337ceb3345367ad855d4aea6dfc81c87"
server_key_version: "encryptionkey@0"
I20260923 17:33:04.860764 26616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:04.892418 26616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:04.896834 26616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:04.897105 26616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:04.898111 26616 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:04.899516 26616 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:04.899595 26616 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:04.899724 26616 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:04.899783 26616 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:04.929057 26616 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:39135
I20260923 17:33:04.929098 26743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:39135 every 8 connection(s)
I20260923 17:33:04.930395 26616 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:33:04.935591 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 26616
I20260923 17:33:04.935784 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:33:04.936443 8311 external_mini_cluster.cc:1428] Setting key 33cc38d1368ddb4a609e87c3e151e3e9
Sep 23 17:33:04 dist-test-slave-p7jt krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184784, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:04.953619 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:04.933219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:59883 (local address 127.8.29.254:36689)
0923 17:33:04.933672 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:04.933677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:04.934925 (+ 1248us) server_negotiation.cc:407] Connection header received
0923 17:33:04.937552 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:04.937558 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:04.937645 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:04.937740 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:04.939716 (+ 1976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:04.940429 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:04.942581 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:04.943032 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:04.943167 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:04.947182 (+ 4015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:04.947190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:04.947195 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:04.947240 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:04.950135 (+ 2895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:04.951029 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:04.951035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:04.951039 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:04.951191 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:04.951708 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:04.951713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:04.951717 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:04.952053 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:04.952187 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:04.953039 (+ 852us) server_negotiation.cc:299] Negotiation successful
0923 17:33:04.953268 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":224,"threads_started":1}
I20260923 17:33:04.954627 26747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:04.933829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:36689 (local address 127.8.29.193:59883)
0923 17:33:04.934729 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:04.934773 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:04.937267 (+ 2494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:04.938027 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:04.938036 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:04.938695 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:04.939531 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:04.939555 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:04.940587 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:04.940592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:04.942378 (+ 1786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:04.942392 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:04.943263 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:04.943269 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:04.943544 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:04.944294 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:04.944323 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:04.946893 (+ 2570us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:04.950339 (+ 3446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:04.950346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:04.950361 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:04.950905 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:04.951378 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:04.951382 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:04.951386 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:04.951589 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:04.952177 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:04.952183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:04.952563 (+ 380us) client_negotiation.cc:769] Sending connection context
0923 17:33:04.952887 (+ 324us) client_negotiation.cc:240] Negotiation successful
0923 17:33:04.953338 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":422,"threads_started":1}
I20260923 17:33:04.956112 26744 heartbeater.cc:347] Connected to a master server at 127.8.29.254:36689
I20260923 17:33:04.956423 26744 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:04.957149 26744 heartbeater.cc:511] Master 127.8.29.254:36689 requested a full tablet report, sending...
I20260923 17:33:04.959246 26555 ts_manager.cc:194] Registered new tserver with Master: db0a54a671e34b16919a5935e708157b (127.8.29.193:39135)
I20260923 17:33:04.960708 26555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:59883
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:33:04.995082 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36689
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:05.316354 26750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:05.316640 26750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:05.316763 26750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:05.324158 26750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:05.324291 26750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:05.324431 26750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:33:05.334229 26750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:05.335561 26750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:05.336848 26750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:05.349844 26755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:05.349777 26756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:05.349858 26758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:05.350647 26750 server_base.cc:1034] running on GCE node
I20260923 17:33:05.351297 26750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:05.352293 26750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:05.353489 26750 hybrid_clock.cc:648] HybridClock initialized: now 1790184785353451 us; error 53 us; skew 500 ppm
I20260923 17:33:05.353866 26750 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:05 dist-test-slave-p7jt krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184785, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:05.356952 26750 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:33:05.358628 26750 webserver.cc:460] Webserver started at http://127.8.29.194:38121/ using document root <none> and password file <none>
I20260923 17:33:05.359323 26750 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:05.359433 26750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:05.359722 26750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:05.362905 26750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "62c8b9397c8e413e9e738dfa027018c5"
format_stamp: "Formatted at 2026-09-23 17:33:05 on dist-test-slave-p7jt"
server_key: "16f434240ac5260a9a21b6247a1c4e7d"
server_key_iv: "69cf505ad7eee3be85aecf3cc1d8c7e5"
server_key_version: "encryptionkey@0"
I20260923 17:33:05.363721 26750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "62c8b9397c8e413e9e738dfa027018c5"
format_stamp: "Formatted at 2026-09-23 17:33:05 on dist-test-slave-p7jt"
server_key: "16f434240ac5260a9a21b6247a1c4e7d"
server_key_iv: "69cf505ad7eee3be85aecf3cc1d8c7e5"
server_key_version: "encryptionkey@0"
I20260923 17:33:05.369668 26750 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:33:05.373593 26765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:05.375190 26750 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:33:05.375379 26750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "62c8b9397c8e413e9e738dfa027018c5"
format_stamp: "Formatted at 2026-09-23 17:33:05 on dist-test-slave-p7jt"
server_key: "16f434240ac5260a9a21b6247a1c4e7d"
server_key_iv: "69cf505ad7eee3be85aecf3cc1d8c7e5"
server_key_version: "encryptionkey@0"
I20260923 17:33:05.375593 26750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:05.400499 26750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:05.405159 26750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:05.405419 26750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:05.406370 26750 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:05.407714 26750 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:05.407793 26750 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:05.407882 26750 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:05.407923 26750 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:05.438534 26750 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:43741
I20260923 17:33:05.438585 26877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:43741 every 8 connection(s)
I20260923 17:33:05.439808 26750 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:33:05.449610 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 26750
I20260923 17:33:05.449800 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:33:05.450348 8311 external_mini_cluster.cc:1428] Setting key 3cde1e0e20ef0c20b00b9c0e50366457
Sep 23 17:33:05 dist-test-slave-p7jt krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184785, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:05.461107 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:05.442424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:38621 (local address 127.8.29.254:36689)
0923 17:33:05.442656 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:05.442662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:05.443838 (+ 1176us) server_negotiation.cc:407] Connection header received
0923 17:33:05.445671 (+ 1833us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:05.445676 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:05.445754 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:05.445845 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:05.447406 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.448144 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.450082 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.450656 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.450778 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:05.454968 (+ 4190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:05.454974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:05.454977 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:05.455015 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:05.457584 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:05.458544 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:05.458548 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:05.458550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:05.458690 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:05.459206 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:05.459211 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:05.459213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:05.459515 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:05.459648 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:05.460531 (+ 883us) server_negotiation.cc:299] Negotiation successful
0923 17:33:05.460752 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260923 17:33:05.462239 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:05.442963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:36689 (local address 127.8.29.194:38621)
0923 17:33:05.443670 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:05.443712 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:05.445384 (+ 1672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:05.446043 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:05.446052 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:05.446635 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:05.447229 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:05.447254 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.448307 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.448312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:05.449921 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:05.449936 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.450889 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.450898 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:05.451183 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:05.451953 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:05.451976 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:05.454697 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:05.457740 (+ 3043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:05.457750 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:05.457771 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:05.458363 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:05.458800 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:05.458804 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:05.458807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:05.459049 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:05.459646 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:05.459654 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:05.460109 (+ 455us) client_negotiation.cc:769] Sending connection context
0923 17:33:05.460418 (+ 309us) client_negotiation.cc:240] Negotiation successful
0923 17:33:05.460996 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":272,"threads_started":1}
I20260923 17:33:05.463817 26878 heartbeater.cc:347] Connected to a master server at 127.8.29.254:36689
I20260923 17:33:05.464134 26878 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:05.464938 26878 heartbeater.cc:511] Master 127.8.29.254:36689 requested a full tablet report, sending...
I20260923 17:33:05.466511 26555 ts_manager.cc:194] Registered new tserver with Master: 62c8b9397c8e413e9e738dfa027018c5 (127.8.29.194:43741)
I20260923 17:33:05.467382 26555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:38621
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:33:05.505374 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36689
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:05.825006 26883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:05.825325 26883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:05.825477 26883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:05.833264 26883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:05.833403 26883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:05.833570 26883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:33:05.844190 26883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:36689
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:05.845600 26883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:05.847013 26883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:05.860356 26891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:05.860424 26889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:05.860446 26888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:05.864068 26883 server_base.cc:1034] running on GCE node
I20260923 17:33:05.864732 26883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:05.865819 26883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:05.867031 26883 hybrid_clock.cc:648] HybridClock initialized: now 1790184785866979 us; error 43 us; skew 500 ppm
I20260923 17:33:05.867362 26883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:05 dist-test-slave-p7jt krb5kdc[26510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184785, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:05.870540 26883 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:33:05.872130 26883 webserver.cc:460] Webserver started at http://127.8.29.195:45119/ using document root <none> and password file <none>
I20260923 17:33:05.872862 26883 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:05.872967 26883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:05.873234 26883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:05.876281 26883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "f98758b117904ebba8949d6467c3ef60"
format_stamp: "Formatted at 2026-09-23 17:33:05 on dist-test-slave-p7jt"
server_key: "1c59c3fdfa501e6fcd53761a7718ea48"
server_key_iv: "3f2ebe3f5a368842a85d9d845ce5ac63"
server_key_version: "encryptionkey@0"
I20260923 17:33:05.877048 26883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "f98758b117904ebba8949d6467c3ef60"
format_stamp: "Formatted at 2026-09-23 17:33:05 on dist-test-slave-p7jt"
server_key: "1c59c3fdfa501e6fcd53761a7718ea48"
server_key_iv: "3f2ebe3f5a368842a85d9d845ce5ac63"
server_key_version: "encryptionkey@0"
I20260923 17:33:05.882596 26883 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260923 17:33:05.886216 26898 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:05.887709 26883 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:05.887869 26883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "f98758b117904ebba8949d6467c3ef60"
format_stamp: "Formatted at 2026-09-23 17:33:05 on dist-test-slave-p7jt"
server_key: "1c59c3fdfa501e6fcd53761a7718ea48"
server_key_iv: "3f2ebe3f5a368842a85d9d845ce5ac63"
server_key_version: "encryptionkey@0"
I20260923 17:33:05.888082 26883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:05.903083 26883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:05.907809 26883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:05.908113 26883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:05.909116 26883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:05.910513 26883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:05.910590 26883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:05.910706 26883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:05.910768 26883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:05.941047 26883 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:35421
I20260923 17:33:05.941090 27010 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:35421 every 8 connection(s)
I20260923 17:33:05.942353 26883 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:33:05.948452 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 26883
I20260923 17:33:05.948621 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:33:05.949072 8311 external_mini_cluster.cc:1428] Setting key 3673e9d7d07a3445e7795c305d32c062
Sep 23 17:33:05 dist-test-slave-p7jt krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184785, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:05.962193 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:05.944941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:53785 (local address 127.8.29.254:36689)
0923 17:33:05.945136 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:05.945141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:05.946260 (+ 1119us) server_negotiation.cc:407] Connection header received
0923 17:33:05.948152 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:05.948158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:05.948243 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:05.948329 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:05.950194 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.951219 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.952888 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.953336 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.953430 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:05.956800 (+ 3370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:05.956806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:05.956810 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:05.956845 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:05.959196 (+ 2351us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:05.960005 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:05.960009 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:05.960011 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:05.960146 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:05.960542 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:05.960546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:05.960548 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:05.960820 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:05.960930 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:05.961721 (+ 791us) server_negotiation.cc:299] Negotiation successful
0923 17:33:05.961923 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:33:05.963534 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:05.945429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:36689 (local address 127.8.29.195:53785)
0923 17:33:05.946112 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:05.946154 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:05.947885 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:05.948573 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:05.948585 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:05.949223 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:05.950008 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:05.950033 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.951356 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.951360 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:05.952733 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:05.952745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.953508 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.953513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:05.953707 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:05.954367 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:05.954391 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:05.956572 (+ 2181us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:05.959419 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:05.959434 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:05.959452 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:05.959881 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:05.960231 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:05.960234 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:05.960237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:05.960447 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:05.960917 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:05.960923 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:05.961259 (+ 336us) client_negotiation.cc:769] Sending connection context
0923 17:33:05.961616 (+ 357us) client_negotiation.cc:240] Negotiation successful
0923 17:33:05.962103 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":263,"threads_started":1}
I20260923 17:33:05.963856 26744 heartbeater.cc:503] Master 127.8.29.254:36689 was elected leader, sending a full tablet report...
I20260923 17:33:05.965161 27011 heartbeater.cc:347] Connected to a master server at 127.8.29.254:36689
I20260923 17:33:05.965504 27011 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:05.966235 27011 heartbeater.cc:511] Master 127.8.29.254:36689 requested a full tablet report, sending...
I20260923 17:33:05.967621 26555 ts_manager.cc:194] Registered new tserver with Master: f98758b117904ebba8949d6467c3ef60 (127.8.29.195:35421)
I20260923 17:33:05.968447 26555 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:53785
I20260923 17:33:05.977181 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:33:05.994285 26746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:05.981367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37926 (local address 127.8.29.254:36689)
0923 17:33:05.981530 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:05.981534 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:05.981981 (+ 447us) server_negotiation.cc:407] Connection header received
0923 17:33:05.982146 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:05.982150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:05.982211 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:05.982289 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:05.983814 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.984556 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.985859 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:05.986252 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:05.986349 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:05.988983 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:05.988988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:05.988992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:05.989025 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:05.991544 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:05.992436 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:05.992442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:05.992446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:05.992599 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:05.993077 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:05.993082 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:05.993084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:05.993422 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:05.993638 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:05.993901 (+ 263us) server_negotiation.cc:299] Negotiation successful
0923 17:33:05.994042 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:33:05.998221 26555 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:33:06.004966 26555 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:33:06.028173 27023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.017011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:43741 (local address 127.0.0.1:46098)
0923 17:33:06.017820 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.017901 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:06.018121 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:06.020090 (+ 1969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:06.020097 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:06.020127 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:06.020580 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.020602 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.022745 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.022751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.025101 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.025118 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.027064 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.027072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.027286 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.027373 (+ 87us) client_negotiation.cc:769] Sending connection context
0923 17:33:06.027491 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 17:33:06.027695 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":487,"mutex_wait_us":3,"thread_start_us":305,"threads_started":1}
I20260923 17:33:06.028528 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.017235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46098 (local address 127.8.29.194:43741)
0923 17:33:06.019428 (+ 2193us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:06.019437 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:06.019466 (+ 29us) server_negotiation.cc:407] Connection header received
0923 17:33:06.019563 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:06.019570 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:06.019847 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:06.020810 (+ 963us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:06.021072 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.022593 (+ 1521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.025285 (+ 2692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.026791 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.027745 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.027963 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:06.028036 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 17:33:06.028210 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2037,"thread_start_us":1936,"threads_started":1}
I20260923 17:33:06.028712 27026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.017789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:35421 (local address 127.0.0.1:51570)
0923 17:33:06.018848 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.018867 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:06.018979 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:06.021664 (+ 2685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:06.021668 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:06.021693 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:06.022053 (+ 360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.022065 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.024065 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.024070 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.025894 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.025906 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.027855 (+ 1949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.027865 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.028038 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.028070 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:33:06.028143 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 17:33:06.028283 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"mutex_wait_us":1,"thread_start_us":2739,"threads_started":1}
I20260923 17:33:06.029278 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.018178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.0.0.1:36038)
0923 17:33:06.020799 (+ 2621us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.020816 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:06.020922 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:06.022045 (+ 1123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:06.022050 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:06.022072 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:06.022392 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.022403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.024194 (+ 1791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.024202 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.025994 (+ 1792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.026009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.028177 (+ 2168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.028183 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.028473 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.028506 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 17:33:06.028838 (+ 332us) client_negotiation.cc:240] Negotiation successful
0923 17:33:06.028988 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2495,"thread_start_us":1215,"threads_started":1}
I20260923 17:33:06.029843 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.018713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51570 (local address 127.8.29.195:35421)
0923 17:33:06.021131 (+ 2418us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:06.021140 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:06.021176 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:33:06.021268 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:06.021276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:06.021488 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:06.021652 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:06.022284 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.023923 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.026059 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.027599 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.029029 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.029255 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:06.029330 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 17:33:06.029518 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2249,"thread_start_us":2117,"threads_started":1}
I20260923 17:33:06.029873 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.019164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36038 (local address 127.8.29.193:39135)
0923 17:33:06.021474 (+ 2310us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:06.021482 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:06.021509 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:33:06.021599 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:06.021606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:06.021838 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:06.022035 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:06.022606 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.024063 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.026167 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.027633 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.027767 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.028013 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:06.028737 (+ 724us) server_negotiation.cc:299] Negotiation successful
0923 17:33:06.029584 (+ 847us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2140,"thread_start_us":1984,"threads_started":1}
I20260923 17:33:06.032454 26813 tablet_service.cc:1467] Processing CreateTablet for tablet 95b8da0e94884e1f89d853f45801b440 (DEFAULT_TABLE table=test-table [id=a6572d67ee7042b58476c90cde0d8cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:06.033689 26813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b8da0e94884e1f89d853f45801b440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:06.033633 26946 tablet_service.cc:1467] Processing CreateTablet for tablet 95b8da0e94884e1f89d853f45801b440 (DEFAULT_TABLE table=test-table [id=a6572d67ee7042b58476c90cde0d8cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:06.033634 26679 tablet_service.cc:1467] Processing CreateTablet for tablet 95b8da0e94884e1f89d853f45801b440 (DEFAULT_TABLE table=test-table [id=a6572d67ee7042b58476c90cde0d8cbe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:06.035171 26946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b8da0e94884e1f89d853f45801b440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:06.035185 26679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95b8da0e94884e1f89d853f45801b440. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:06.046289 27029 tablet_bootstrap.cc:492] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: Bootstrap starting.
I20260923 17:33:06.050272 27029 tablet_bootstrap.cc:654] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:06.051800 27029 log.cc:824] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:06.053455 27031 tablet_bootstrap.cc:492] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: Bootstrap starting.
I20260923 17:33:06.058858 27031 tablet_bootstrap.cc:654] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:06.060428 27031 log.cc:824] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:06.061785 27030 tablet_bootstrap.cc:492] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b: Bootstrap starting.
I20260923 17:33:06.065129 27030 tablet_bootstrap.cc:654] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:06.066262 27030 log.cc:824] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:06.066916 27029 tablet_bootstrap.cc:492] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: No bootstrap required, opened a new log
I20260923 17:33:06.066912 27031 tablet_bootstrap.cc:492] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: No bootstrap required, opened a new log
I20260923 17:33:06.067224 27031 ts_tablet_manager.cc:1397] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.004s
I20260923 17:33:06.067224 27029 ts_tablet_manager.cc:1397] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.011s
I20260923 17:33:06.068988 27030 tablet_bootstrap.cc:492] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b: No bootstrap required, opened a new log
I20260923 17:33:06.069245 27030 ts_tablet_manager.cc:1397] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260923 17:33:06.076915 27030 raft_consensus.cc:348] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } }
I20260923 17:33:06.077335 27030 raft_consensus.cc:374] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:06.077427 27030 raft_consensus.cc:729] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db0a54a671e34b16919a5935e708157b, State: Initialized, Role: FOLLOWER
I20260923 17:33:06.077744 27029 raft_consensus.cc:348] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } }
I20260923 17:33:06.078030 27030 consensus_queue.cc:260] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } }
I20260923 17:33:06.078207 27029 raft_consensus.cc:374] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:06.078296 27029 raft_consensus.cc:729] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62c8b9397c8e413e9e738dfa027018c5, State: Initialized, Role: FOLLOWER
I20260923 17:33:06.078349 27031 raft_consensus.cc:348] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } }
I20260923 17:33:06.078792 27031 raft_consensus.cc:374] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:06.078886 27031 raft_consensus.cc:729] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f98758b117904ebba8949d6467c3ef60, State: Initialized, Role: FOLLOWER
I20260923 17:33:06.078935 27029 consensus_queue.cc:260] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } }
I20260923 17:33:06.079656 27031 consensus_queue.cc:260] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } }
I20260923 17:33:06.080051 27030 ts_tablet_manager.cc:1428] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
I20260923 17:33:06.080940 26878 heartbeater.cc:503] Master 127.8.29.254:36689 was elected leader, sending a full tablet report...
I20260923 17:33:06.081698 27029 ts_tablet_manager.cc:1428] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: Time spent starting tablet: real 0.014s user 0.013s sys 0.001s
I20260923 17:33:06.082815 27011 heartbeater.cc:503] Master 127.8.29.254:36689 was elected leader, sending a full tablet report...
I20260923 17:33:06.083455 27031 ts_tablet_manager.cc:1428] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
I20260923 17:33:06.181205 27037 raft_consensus.cc:482] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:33:06.181515 27037 raft_consensus.cc:504] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } }
W20260923 17:33:06.182793 26745 tablet.cc:2369] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:06.183422 27037 leader_election.cc:288] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 62c8b9397c8e413e9e738dfa027018c5 (127.8.29.194:43741), db0a54a671e34b16919a5935e708157b (127.8.29.193:39135)
I20260923 17:33:06.190057 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.183902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:42953 (local address 127.8.29.194:43741)
0923 17:33:06.184113 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:06.184119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:06.184137 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:33:06.184196 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:06.184200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:06.184257 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:06.184344 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:06.184955 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.185899 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.188156 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.189232 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.189399 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.189476 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:06.189690 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 17:33:06.189822 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:33:06.190093 27013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.183666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:43741 (local address 127.8.29.195:42953)
0923 17:33:06.183855 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.183874 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:06.183993 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:06.184519 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:06.184525 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:06.184565 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:06.184779 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.184792 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.186041 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.186045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.187976 (+ 1931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.187991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.189393 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.189397 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.189539 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.189565 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:33:06.189639 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 17:33:06.189740 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":1}
I20260923 17:33:06.190650 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.184306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:58679 (local address 127.8.29.193:39135)
0923 17:33:06.184504 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:06.184510 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:06.184837 (+ 327us) server_negotiation.cc:407] Connection header received
0923 17:33:06.184964 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:06.184968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:06.185050 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:06.185154 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:06.185609 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.186448 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.188560 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.189718 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.189869 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.189940 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:06.190318 (+ 378us) server_negotiation.cc:299] Negotiation successful
0923 17:33:06.190473 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:33:06.191180 27038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.184071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.8.29.195:58679)
0923 17:33:06.184712 (+ 641us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.184729 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:06.184842 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:06.185165 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:06.185168 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:06.185189 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:06.185478 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.185485 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.186616 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.186621 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.188410 (+ 1789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:06.188420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.189919 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.189925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:06.190113 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.190145 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:33:06.190825 (+ 680us) client_negotiation.cc:240] Negotiation successful
0923 17:33:06.190961 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":331,"threads_started":1}
I20260923 17:33:06.191162 26833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b8da0e94884e1f89d853f45801b440" candidate_uuid: "f98758b117904ebba8949d6467c3ef60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c8b9397c8e413e9e738dfa027018c5" is_pre_election: true
W20260923 17:33:06.191706 26879 tablet.cc:2369] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:06.191803 26833 raft_consensus.cc:2454] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f98758b117904ebba8949d6467c3ef60 in term 0.
I20260923 17:33:06.192238 26699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b8da0e94884e1f89d853f45801b440" candidate_uuid: "f98758b117904ebba8949d6467c3ef60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0a54a671e34b16919a5935e708157b" is_pre_election: true
I20260923 17:33:06.192744 26902 leader_election.cc:302] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c8b9397c8e413e9e738dfa027018c5, f98758b117904ebba8949d6467c3ef60; no voters:
I20260923 17:33:06.192821 26699 raft_consensus.cc:2454] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f98758b117904ebba8949d6467c3ef60 in term 0.
I20260923 17:33:06.193243 27037 raft_consensus.cc:2781] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:33:06.193419 27037 raft_consensus.cc:482] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:33:06.193528 27037 raft_consensus.cc:3037] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 0 FOLLOWER]: Advancing to term 1
W20260923 17:33:06.193846 27012 tablet.cc:2369] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:06.196153 27037 raft_consensus.cc:504] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } }
I20260923 17:33:06.196730 27037 leader_election.cc:288] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [CANDIDATE]: Term 1 election: Requested vote from peers 62c8b9397c8e413e9e738dfa027018c5 (127.8.29.194:43741), db0a54a671e34b16919a5935e708157b (127.8.29.193:39135)
I20260923 17:33:06.197283 26833 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b8da0e94884e1f89d853f45801b440" candidate_uuid: "f98758b117904ebba8949d6467c3ef60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62c8b9397c8e413e9e738dfa027018c5"
I20260923 17:33:06.197382 26699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95b8da0e94884e1f89d853f45801b440" candidate_uuid: "f98758b117904ebba8949d6467c3ef60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db0a54a671e34b16919a5935e708157b"
I20260923 17:33:06.197474 26833 raft_consensus.cc:3037] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:06.197611 26699 raft_consensus.cc:3037] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:06.199769 26833 raft_consensus.cc:2454] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f98758b117904ebba8949d6467c3ef60 in term 1.
I20260923 17:33:06.199892 26699 raft_consensus.cc:2454] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f98758b117904ebba8949d6467c3ef60 in term 1.
I20260923 17:33:06.200253 26902 leader_election.cc:302] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62c8b9397c8e413e9e738dfa027018c5, f98758b117904ebba8949d6467c3ef60; no voters:
I20260923 17:33:06.200551 27037 raft_consensus.cc:2781] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:06.200994 27037 raft_consensus.cc:686] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 1 LEADER]: Becoming Leader. State: Replica: f98758b117904ebba8949d6467c3ef60, State: Running, Role: LEADER
I20260923 17:33:06.201576 27037 consensus_queue.cc:237] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } }
I20260923 17:33:06.206699 26552 catalog_manager.cc:5495] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 reported cstate change: term changed from 0 to 1, leader changed from <none> to f98758b117904ebba8949d6467c3ef60 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "f98758b117904ebba8949d6467c3ef60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f98758b117904ebba8949d6467c3ef60" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35421 } health_report { overall_health: HEALTHY } } }
I20260923 17:33:06.226588 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.220978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51584 (local address 127.8.29.195:35421)
0923 17:33:06.221160 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:06.221164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:06.221226 (+ 62us) server_negotiation.cc:407] Connection header received
0923 17:33:06.221418 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:06.221422 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:06.221505 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:06.221637 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:06.221951 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.222719 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.224168 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.224606 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.224710 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.225312 (+ 602us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:06.225964 (+ 652us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:06.226105 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:06.226288 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 17:33:06.226335 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:33:06.235359 26833 raft_consensus.cc:1261] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [term 1 FOLLOWER]: Refusing update from remote peer f98758b117904ebba8949d6467c3ef60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:06.235359 26699 raft_consensus.cc:1261] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [term 1 FOLLOWER]: Refusing update from remote peer f98758b117904ebba8949d6467c3ef60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:06.236502 27039 consensus_queue.cc:1035] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62c8b9397c8e413e9e738dfa027018c5" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 43741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:06.236963 27037 consensus_queue.cc:1035] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db0a54a671e34b16919a5935e708157b" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 39135 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:06.271497 26552 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:06.271852 26552 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:06.276101 26552 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f64d9ddb8db4825ac3dcfc52afb6732: Sending DeleteTablet for 3 replicas of tablet 95b8da0e94884e1f89d853f45801b440
I20260923 17:33:06.277581 26946 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b8da0e94884e1f89d853f45801b440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:06 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:51570
I20260923 17:33:06.277572 26813 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b8da0e94884e1f89d853f45801b440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:06 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:46098
I20260923 17:33:06.277695 26679 tablet_service.cc:1514] Processing DeleteTablet for tablet 95b8da0e94884e1f89d853f45801b440 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:06 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:36038
I20260923 17:33:06.284783 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 26616
I20260923 17:33:06.295888 27053 tablet_replica.cc:330] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: stopping tablet replica
I20260923 17:33:06.296464 27053 raft_consensus.cc:2229] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:06.297569 27053 raft_consensus.cc:2258] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:06.300258 27053 ts_tablet_manager.cc:1905] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:06.307119 27054 tablet_replica.cc:330] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b: stopping tablet replica
I20260923 17:33:06.307667 27054 raft_consensus.cc:2229] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:06.308264 27054 raft_consensus.cc:2258] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:06.309085 27052 tablet_replica.cc:330] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: stopping tablet replica
I20260923 17:33:06.309146 27053 ts_tablet_manager.cc:1918] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:06.309443 27053 log.cc:1196] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/95b8da0e94884e1f89d853f45801b440
I20260923 17:33:06.309510 27052 raft_consensus.cc:2229] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:33:06.310112 27052 raft_consensus.cc:2258] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:06.310141 27053 ts_tablet_manager.cc:1939] T 95b8da0e94884e1f89d853f45801b440 P 62c8b9397c8e413e9e738dfa027018c5: Deleting consensus metadata
I20260923 17:33:06.312304 26542 catalog_manager.cc:4841] TS 62c8b9397c8e413e9e738dfa027018c5 (127.8.29.194:43741): tablet 95b8da0e94884e1f89d853f45801b440 (table test-table [id=a6572d67ee7042b58476c90cde0d8cbe]) successfully deleted
I20260923 17:33:06.314805 27052 ts_tablet_manager.cc:1905] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 23 17:33:06 dist-test-slave-p7jt krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:33:06.322506 27054 ts_tablet_manager.cc:1905] T 95b8da0e94884e1f89d853f45801b440 P db0a54a671e34b16919a5935e708157b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:06.325145 27028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.286634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36040 (local address 127.8.29.193:39135)
0923 17:33:06.286833 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:06.286839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:06.294204 (+ 7365us) server_negotiation.cc:407] Connection header received
0923 17:33:06.294451 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:06.294478 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:06.294603 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:06.294720 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:06.299801 (+ 5081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.300994 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.304970 (+ 3976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.305689 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.305801 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.316902 (+ 11101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:06.316923 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:06.316942 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:06.317002 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:06.320921 (+ 3919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:06.322059 (+ 1138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:06.322072 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:06.322076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:06.322268 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:06.322944 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:06.322950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:06.322954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:06.323499 (+ 545us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:06.323829 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:06.324470 (+ 641us) server_negotiation.cc:299] Negotiation successful
0923 17:33:06.324720 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:33:06.327256 26739 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:06.322405 27052 ts_tablet_manager.cc:1918] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:06.328450 27052 log.cc:1196] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/95b8da0e94884e1f89d853f45801b440
I20260923 17:33:06.329082 27052 ts_tablet_manager.cc:1939] T 95b8da0e94884e1f89d853f45801b440 P f98758b117904ebba8949d6467c3ef60: Deleting consensus metadata
I20260923 17:33:06.331223 26540 catalog_manager.cc:4841] TS f98758b117904ebba8949d6467c3ef60 (127.8.29.195:35421): tablet 95b8da0e94884e1f89d853f45801b440 (table test-table [id=a6572d67ee7042b58476c90cde0d8cbe]) successfully deleted
I20260923 17:33:06.421766 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 26616
I20260923 17:33:06.443043 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 26750
W20260923 17:33:06.442987 26542 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.29.193:39135 (error 108)
I20260923 17:33:06.445454 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.444898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.0.0.1:36054)
0923 17:33:06.445178 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.445290 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:39135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260923 17:33:06.445737 26542 catalog_manager.cc:4584] TS db0a54a671e34b16919a5935e708157b (127.8.29.193:39135): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 95b8da0e94884e1f89d853f45801b440: Network error: Client connection negotiation failed: client connection to 127.8.29.193:39135: connect: Connection refused (error 111)
Sep 23 17:33:06 dist-test-slave-p7jt krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:33:06.463428 27024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.444411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46104 (local address 127.8.29.194:43741)
0923 17:33:06.444591 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:06.444597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:06.444615 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:33:06.444675 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:06.444705 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:06.444800 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:06.444912 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:06.446596 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.447722 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.449726 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.450346 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.450596 (+ 250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.456163 (+ 5567us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:06.456181 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:06.456200 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:06.456247 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:06.460243 (+ 3996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:06.461022 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:06.461033 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:06.461037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:06.461199 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:06.461633 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:06.461638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:06.461641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:06.462168 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:06.462508 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:06.462797 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 17:33:06.463045 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 17:33:06.464109 26873 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:06.499571 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.498984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.0.0.1:36060)
0923 17:33:06.499244 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.499374 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:39135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 17:33:06.500897 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.500434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.0.0.1:36072)
0923 17:33:06.500612 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.500709 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:39135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:33:06.559155 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 26750
I20260923 17:33:06.561843 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.561353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.0.0.1:36088)
0923 17:33:06.561574 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.561671 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:39135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 17:33:06.562891 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.562552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.0.0.1:36098)
0923 17:33:06.562682 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.562766 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:39135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 17:33:06.580322 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 26883
Sep 23 17:33:06 dist-test-slave-p7jt krb5kdc[26510](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184783, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:33:06.599074 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.581259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.8.29.195:35421)
0923 17:33:06.581418 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:06.581422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:06.581444 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:33:06.581571 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:06.581575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:06.581646 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:06.581726 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:06.583552 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.584616 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.586531 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:06.587012 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:06.587113 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:06.592692 (+ 5579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:06.592730 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:06.592750 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:06.592798 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:06.596033 (+ 3235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:06.596900 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:06.596908 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:06.596910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:06.597109 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:06.597549 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:06.597553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:06.597556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:06.597959 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:06.598209 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:06.598593 (+ 384us) server_negotiation.cc:299] Negotiation successful
0923 17:33:06.598765 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:33:06.599838 27006 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:06.644229 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.643715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.0.0.1:36112)
0923 17:33:06.643948 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.644049 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:39135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260923 17:33:06.645372 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.644917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.0.0.1:36116)
0923 17:33:06.645138 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.645218 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:39135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260923 17:33:06.723773 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 26883
I20260923 17:33:06.740032 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 26523
I20260923 17:33:06.741110 26585 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:06.851763 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 26523
I20260923 17:33:06.851812 27025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:06.851261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:39135 (local address 127.0.0.1:59258)
0923 17:33:06.851515 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:06.851619 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:39135: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
2026-09-23T17:33:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4289 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T17:33:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:06Z Disabled control of system clock
I20260923 17:33:06.924559 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:33951
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:33951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:33:07.258315 27064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:07.258636 27064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:07.258728 27064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:07.266050 27064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:07.266175 27064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:07.266219 27064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:07.266273 27064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:07.266316 27064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:07.275777 27064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:33951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:33951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:07.277158 27064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:07.278968 27064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:07.291000 27069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:07.291137 27072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:07.291091 27070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:07.292291 27064 server_base.cc:1034] running on GCE node
I20260923 17:33:07.293064 27064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:07.294559 27064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:07.295779 27064 hybrid_clock.cc:648] HybridClock initialized: now 1790184787295742 us; error 62 us; skew 500 ppm
I20260923 17:33:07.296103 27064 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:07.302121 27064 webserver.cc:460] Webserver started at http://127.8.29.254:36733/ using document root <none> and password file <none>
I20260923 17:33:07.302922 27064 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:07.303027 27064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:07.303323 27064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:07.306393 27064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "42da56e783b9474fad9246fa3c10b400"
format_stamp: "Formatted at 2026-09-23 17:33:07 on dist-test-slave-p7jt"
server_key: "e1b3c22c6005b556e896293ae95507f4"
server_key_iv: "95d592e74696aaa3ccf614d8a28ce218"
server_key_version: "encryptionkey@0"
I20260923 17:33:07.307291 27064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "42da56e783b9474fad9246fa3c10b400"
format_stamp: "Formatted at 2026-09-23 17:33:07 on dist-test-slave-p7jt"
server_key: "e1b3c22c6005b556e896293ae95507f4"
server_key_iv: "95d592e74696aaa3ccf614d8a28ce218"
server_key_version: "encryptionkey@0"
I20260923 17:33:07.312862 27064 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:33:07.316642 27078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:07.318234 27064 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:33:07.318434 27064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "42da56e783b9474fad9246fa3c10b400"
format_stamp: "Formatted at 2026-09-23 17:33:07 on dist-test-slave-p7jt"
server_key: "e1b3c22c6005b556e896293ae95507f4"
server_key_iv: "95d592e74696aaa3ccf614d8a28ce218"
server_key_version: "encryptionkey@0"
I20260923 17:33:07.318688 27064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:07.340518 27064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:07.341506 27064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:07.341810 27064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:07.359928 27064 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:33951
I20260923 17:33:07.359964 27129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:33951 every 8 connection(s)
I20260923 17:33:07.361285 27064 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:07.365147 27130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:07.368567 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 27064
I20260923 17:33:07.368739 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:07.369287 8311 external_mini_cluster.cc:1428] Setting key cb99e8064a2f9f7cc2bc0310c37f2dde
I20260923 17:33:07.378021 27130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400: Bootstrap starting.
I20260923 17:33:07.381492 27130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:07.382872 27130 log.cc:824] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:07.383855 27133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:07.371432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48076 (local address 127.8.29.254:33951)
0923 17:33:07.372538 (+ 1106us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:07.372550 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:07.372594 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:33:07.374560 (+ 1966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:07.374590 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:07.375145 (+ 555us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:07.375562 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:07.376191 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:07.377458 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:07.378918 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:07.379635 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:07.379820 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:07.381001 (+ 1181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:07.381010 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:07.381026 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:07.381607 (+ 581us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:07.381767 (+ 160us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:07.381796 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:07.381922 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:07.382191 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 17:33:07.382655 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":638,"thread_start_us":405,"threads_started":1}
I20260923 17:33:07.386206 27130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400: No bootstrap required, opened a new log
I20260923 17:33:07.394069 27130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42da56e783b9474fad9246fa3c10b400" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33951 } }
I20260923 17:33:07.394536 27130 raft_consensus.cc:374] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:07.394649 27130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42da56e783b9474fad9246fa3c10b400, State: Initialized, Role: FOLLOWER
I20260923 17:33:07.395260 27130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42da56e783b9474fad9246fa3c10b400" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33951 } }
I20260923 17:33:07.395506 27130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:07.395643 27130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:07.395798 27130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:07.398306 27130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42da56e783b9474fad9246fa3c10b400" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33951 } }
I20260923 17:33:07.398936 27130 leader_election.cc:302] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 42da56e783b9474fad9246fa3c10b400; no voters:
I20260923 17:33:07.399540 27130 leader_election.cc:288] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:07.399703 27135 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:07.400485 27135 raft_consensus.cc:686] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [term 1 LEADER]: Becoming Leader. State: Replica: 42da56e783b9474fad9246fa3c10b400, State: Running, Role: LEADER
I20260923 17:33:07.401048 27135 consensus_queue.cc:237] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42da56e783b9474fad9246fa3c10b400" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33951 } }
I20260923 17:33:07.401666 27130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:07.403846 27137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 42da56e783b9474fad9246fa3c10b400. Latest consensus state: current_term: 1 leader_uuid: "42da56e783b9474fad9246fa3c10b400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42da56e783b9474fad9246fa3c10b400" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33951 } } }
I20260923 17:33:07.404521 27137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:07.404098 27136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "42da56e783b9474fad9246fa3c10b400" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "42da56e783b9474fad9246fa3c10b400" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 33951 } } }
I20260923 17:33:07.406229 27136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:07.407593 27144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:07.413247 27144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:07.423507 27144 catalog_manager.cc:1347] Generated new cluster ID: ddb764af61024774bfe4ab27d1ffff6b
I20260923 17:33:07.423640 27144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:07.448895 27144 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:07.450249 27144 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:07.459247 27144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400: Generated new TSK 0
I20260923 17:33:07.460161 27144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:07.469825 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33951
--builtin_ntp_servers=127.8.29.212:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:33:07.828974 27154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:07.829265 27154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:07.829351 27154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:07.836817 27154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:07.836931 27154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:07.837040 27154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:33:07.846285 27154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:07.847636 27154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:07.849452 27154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:07.862910 27159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:07.862901 27162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:07.862938 27160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:07.863740 27154 server_base.cc:1034] running on GCE node
I20260923 17:33:07.864365 27154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:07.865375 27154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:07.866571 27154 hybrid_clock.cc:648] HybridClock initialized: now 1790184787866508 us; error 55 us; skew 500 ppm
I20260923 17:33:07.866950 27154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:07.869307 27154 webserver.cc:460] Webserver started at http://127.8.29.193:36655/ using document root <none> and password file <none>
I20260923 17:33:07.870038 27154 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:07.870141 27154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:07.870486 27154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:07.873569 27154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "176380baba1e42839c4d1112241e3c65"
format_stamp: "Formatted at 2026-09-23 17:33:07 on dist-test-slave-p7jt"
server_key: "32287a96b5c248e0e98f265ae84af444"
server_key_iv: "75db7643db04ff184afbf3d7d40220c0"
server_key_version: "encryptionkey@0"
I20260923 17:33:07.874655 27154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "176380baba1e42839c4d1112241e3c65"
format_stamp: "Formatted at 2026-09-23 17:33:07 on dist-test-slave-p7jt"
server_key: "32287a96b5c248e0e98f265ae84af444"
server_key_iv: "75db7643db04ff184afbf3d7d40220c0"
server_key_version: "encryptionkey@0"
I20260923 17:33:07.880358 27154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:33:07.884248 27168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:07.885761 27154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:33:07.885921 27154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "176380baba1e42839c4d1112241e3c65"
format_stamp: "Formatted at 2026-09-23 17:33:07 on dist-test-slave-p7jt"
server_key: "32287a96b5c248e0e98f265ae84af444"
server_key_iv: "75db7643db04ff184afbf3d7d40220c0"
server_key_version: "encryptionkey@0"
I20260923 17:33:07.886133 27154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:07.915748 27154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:07.916646 27154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:07.916950 27154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:07.917976 27154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:07.919545 27154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:07.919633 27154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:07.919735 27154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:07.919804 27154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:07.950956 27154 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:43357
I20260923 17:33:07.951043 27280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:43357 every 8 connection(s)
I20260923 17:33:07.952266 27154 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:33:07.958926 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 27154
I20260923 17:33:07.959107 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:33:07.959666 8311 external_mini_cluster.cc:1428] Setting key 180250bc9fe862cac3a50c70c260de6e
I20260923 17:33:07.962525 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33951
--builtin_ntp_servers=127.8.29.212:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:33:07.965759 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:07.954942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:50721 (local address 127.8.29.254:33951)
0923 17:33:07.955375 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:07.955380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:07.956665 (+ 1285us) server_negotiation.cc:407] Connection header received
0923 17:33:07.958558 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:07.958563 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:07.958656 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:07.958789 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:07.960365 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:07.961285 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:07.963082 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:07.963486 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:07.963594 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:07.964954 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:07.964959 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:07.964962 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:07.964981 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:07.965017 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:07.965029 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:07.965101 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:07.965321 (+ 220us) server_negotiation.cc:299] Negotiation successful
0923 17:33:07.965420 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20260923 17:33:07.966868 27284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:07.955492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33951 (local address 127.8.29.193:50721)
0923 17:33:07.956496 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:07.956541 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:07.958209 (+ 1668us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:07.959124 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:07.959137 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:07.959297 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:07.960173 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:07.960200 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:07.961443 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:07.961448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:07.962943 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:07.962957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:07.963704 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:07.963711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:07.963938 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:07.964594 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:33:07.964597 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:07.964639 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:33:07.964721 (+ 82us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:07.965117 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:07.965129 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:07.965135 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:33:07.965298 (+ 163us) client_negotiation.cc:240] Negotiation successful
0923 17:33:07.965723 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":336,"threads_started":1}
I20260923 17:33:07.968575 27281 heartbeater.cc:347] Connected to a master server at 127.8.29.254:33951
I20260923 17:33:07.968917 27281 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:07.969784 27281 heartbeater.cc:511] Master 127.8.29.254:33951 requested a full tablet report, sending...
I20260923 17:33:07.972179 27095 ts_manager.cc:194] Registered new tserver with Master: 176380baba1e42839c4d1112241e3c65 (127.8.29.193:43357)
I20260923 17:33:07.973728 27095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.193:50721
W20260923 17:33:08.293849 27285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:08.294155 27285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:08.294282 27285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:08.301709 27285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:08.301821 27285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:08.301963 27285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:33:08.311601 27285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:08.312932 27285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:08.314729 27285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:08.327783 27291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:08.327747 27290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:08.327723 27293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:08.328114 27285 server_base.cc:1034] running on GCE node
I20260923 17:33:08.328950 27285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:08.329912 27285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:08.331154 27285 hybrid_clock.cc:648] HybridClock initialized: now 1790184788331094 us; error 43 us; skew 500 ppm
I20260923 17:33:08.331517 27285 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:08.333833 27285 webserver.cc:460] Webserver started at http://127.8.29.194:37907/ using document root <none> and password file <none>
I20260923 17:33:08.334550 27285 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:08.334653 27285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:08.334959 27285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:08.337841 27285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "baf5daa3e3a747bab63cb3b2e832c273"
format_stamp: "Formatted at 2026-09-23 17:33:08 on dist-test-slave-p7jt"
server_key: "a5dbfd82d5d2d6aa62c4e3e7f9c1e5f0"
server_key_iv: "d9496b6e5eb4814c23a28d4f3cfa722b"
server_key_version: "encryptionkey@0"
I20260923 17:33:08.338667 27285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "baf5daa3e3a747bab63cb3b2e832c273"
format_stamp: "Formatted at 2026-09-23 17:33:08 on dist-test-slave-p7jt"
server_key: "a5dbfd82d5d2d6aa62c4e3e7f9c1e5f0"
server_key_iv: "d9496b6e5eb4814c23a28d4f3cfa722b"
server_key_version: "encryptionkey@0"
I20260923 17:33:08.343827 27285 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 17:33:08.347504 27299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:08.349050 27285 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:33:08.349225 27285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "baf5daa3e3a747bab63cb3b2e832c273"
format_stamp: "Formatted at 2026-09-23 17:33:08 on dist-test-slave-p7jt"
server_key: "a5dbfd82d5d2d6aa62c4e3e7f9c1e5f0"
server_key_iv: "d9496b6e5eb4814c23a28d4f3cfa722b"
server_key_version: "encryptionkey@0"
I20260923 17:33:08.349449 27285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:08.384896 27285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:08.385769 27285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:08.386070 27285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:08.387074 27285 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:08.388520 27285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:08.388612 27285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:08.388720 27285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:08.388787 27285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:08.418690 27285 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:34039
I20260923 17:33:08.418754 27411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:34039 every 8 connection(s)
I20260923 17:33:08.420019 27285 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:33:08.428540 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 27285
I20260923 17:33:08.428730 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:33:08.429293 8311 external_mini_cluster.cc:1428] Setting key 8ff1d7a8fff8fc8048eec9cdd3ebcfda
I20260923 17:33:08.432240 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33951
--builtin_ntp_servers=127.8.29.212:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:33:08.433225 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.422470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:58805 (local address 127.8.29.254:33951)
0923 17:33:08.422687 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:08.422694 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:08.424194 (+ 1500us) server_negotiation.cc:407] Connection header received
0923 17:33:08.425981 (+ 1787us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:08.425986 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:08.426073 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:08.426163 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:08.427367 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.428098 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.430297 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.430817 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.430925 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.432336 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:08.432342 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:08.432347 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:08.432371 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:08.432432 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:08.432451 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:08.432543 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:08.432798 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 17:33:08.432914 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:33:08.434206 27414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.423125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33951 (local address 127.8.29.194:58805)
0923 17:33:08.424043 (+ 918us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:08.424081 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:08.425699 (+ 1618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:08.426438 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:08.426447 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:08.426587 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:08.427202 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.427221 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.428275 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.428280 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.430161 (+ 1881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.430174 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.431054 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.431062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.431306 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.431965 (+ 659us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:33:08.431968 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:08.432011 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:33:08.432107 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:08.432584 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:08.432594 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:08.432601 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:33:08.432781 (+ 180us) client_negotiation.cc:240] Negotiation successful
0923 17:33:08.433117 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":280,"threads_started":1}
I20260923 17:33:08.435751 27412 heartbeater.cc:347] Connected to a master server at 127.8.29.254:33951
I20260923 17:33:08.436087 27412 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:08.436863 27412 heartbeater.cc:511] Master 127.8.29.254:33951 requested a full tablet report, sending...
I20260923 17:33:08.438319 27095 ts_manager.cc:194] Registered new tserver with Master: baf5daa3e3a747bab63cb3b2e832c273 (127.8.29.194:34039)
I20260923 17:33:08.439186 27095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.194:58805
W20260923 17:33:08.763597 27415 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:08.763888 27415 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:08.763975 27415 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:08.771253 27415 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:08.771369 27415 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:08.771476 27415 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:33:08.780659 27415 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:33951
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:08.781893 27415 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:08.783677 27415 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:08.796633 27420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:08.796700 27421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:08.796672 27423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:08.797452 27415 server_base.cc:1034] running on GCE node
I20260923 17:33:08.798105 27415 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:08.799183 27415 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:08.800380 27415 hybrid_clock.cc:648] HybridClock initialized: now 1790184788800320 us; error 60 us; skew 500 ppm
I20260923 17:33:08.800691 27415 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:08.803136 27415 webserver.cc:460] Webserver started at http://127.8.29.195:36495/ using document root <none> and password file <none>
I20260923 17:33:08.803876 27415 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:08.803983 27415 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:08.804296 27415 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:08.807274 27415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "6da1a68f2ee64b25ab4cf2f186ca4589"
format_stamp: "Formatted at 2026-09-23 17:33:08 on dist-test-slave-p7jt"
server_key: "30223feda850bf7a70b340b37e7df85c"
server_key_iv: "17ad45095054f8918b80e402bdb3ad4f"
server_key_version: "encryptionkey@0"
I20260923 17:33:08.808105 27415 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "6da1a68f2ee64b25ab4cf2f186ca4589"
format_stamp: "Formatted at 2026-09-23 17:33:08 on dist-test-slave-p7jt"
server_key: "30223feda850bf7a70b340b37e7df85c"
server_key_iv: "17ad45095054f8918b80e402bdb3ad4f"
server_key_version: "encryptionkey@0"
I20260923 17:33:08.813414 27415 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:33:08.816932 27429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:08.818291 27415 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:33:08.818446 27415 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "6da1a68f2ee64b25ab4cf2f186ca4589"
format_stamp: "Formatted at 2026-09-23 17:33:08 on dist-test-slave-p7jt"
server_key: "30223feda850bf7a70b340b37e7df85c"
server_key_iv: "17ad45095054f8918b80e402bdb3ad4f"
server_key_version: "encryptionkey@0"
I20260923 17:33:08.818686 27415 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:08.839277 27415 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:08.840132 27415 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:08.840415 27415 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:08.841398 27415 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:08.842983 27415 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:08.843078 27415 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:08.843187 27415 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:08.843254 27415 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:08.875281 27415 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:35127
I20260923 17:33:08.875362 27541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:35127 every 8 connection(s)
I20260923 17:33:08.876590 27415 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:33:08.886612 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 27415
I20260923 17:33:08.886773 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:33:08.887184 8311 external_mini_cluster.cc:1428] Setting key 1a0815c7827a95505a996a995457d276
I20260923 17:33:08.888818 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.879144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:51417 (local address 127.8.29.254:33951)
0923 17:33:08.879342 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:08.879347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:08.880776 (+ 1429us) server_negotiation.cc:407] Connection header received
0923 17:33:08.882545 (+ 1769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:08.882550 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:08.882639 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:08.882726 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:08.883796 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.884523 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.886054 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.886495 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.886612 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.887943 (+ 1331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:08.887948 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:08.887952 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:08.887971 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:08.888002 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:08.888014 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:08.888090 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:08.888412 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 17:33:08.888508 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:33:08.889808 27544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.879661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:33951 (local address 127.8.29.195:51417)
0923 17:33:08.880603 (+ 942us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:08.880650 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:08.882264 (+ 1614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:08.882966 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:08.882975 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:08.883064 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:08.883629 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.883645 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.884676 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.884679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.885916 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.885928 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.886708 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.886715 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.886977 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.887615 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:33:08.887617 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:08.887659 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:33:08.887746 (+ 87us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:08.888099 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:08.888109 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:08.888118 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:33:08.888324 (+ 206us) client_negotiation.cc:240] Negotiation successful
0923 17:33:08.888690 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":286,"threads_started":1}
I20260923 17:33:08.891283 27542 heartbeater.cc:347] Connected to a master server at 127.8.29.254:33951
I20260923 17:33:08.891566 27542 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:08.892308 27542 heartbeater.cc:511] Master 127.8.29.254:33951 requested a full tablet report, sending...
I20260923 17:33:08.893723 27095 ts_manager.cc:194] Registered new tserver with Master: 6da1a68f2ee64b25ab4cf2f186ca4589 (127.8.29.195:35127)
I20260923 17:33:08.894536 27095 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.195:51417
I20260923 17:33:08.902385 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:33:08.912230 27283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.906385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48078 (local address 127.8.29.254:33951)
0923 17:33:08.906643 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:08.906647 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:08.906994 (+ 347us) server_negotiation.cc:407] Connection header received
0923 17:33:08.907123 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:08.907127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:08.907203 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:08.907307 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:08.907801 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.908594 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.909973 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.910430 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.910592 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.911573 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:08.911579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:08.911582 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:08.911602 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:08.911636 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:08.911651 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:08.911796 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:08.911909 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 17:33:08.911960 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260923 17:33:08.916211 27095 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:33:08.923293 27095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:33:08.947001 27554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.935223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:35127 (local address 127.0.0.1:57512)
0923 17:33:08.937236 (+ 2013us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:08.937266 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:08.937454 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:08.938559 (+ 1105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:08.938566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:08.938813 (+ 247us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:08.939952 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.939966 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.941830 (+ 1864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.941835 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.944168 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.944181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.945984 (+ 1803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.945991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.946183 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.946241 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 17:33:08.946363 (+ 122us) client_negotiation.cc:240] Negotiation successful
0923 17:33:08.946604 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1797,"mutex_wait_us":234,"thread_start_us":1689,"threads_started":1}
I20260923 17:33:08.947580 27559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.938223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.8.29.194:34039)
0923 17:33:08.939712 (+ 1489us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:08.939721 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:08.940258 (+ 537us) server_negotiation.cc:407] Connection header received
0923 17:33:08.940397 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:08.940403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:08.940572 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:08.940725 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:08.941236 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.942685 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.945179 (+ 2494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.946260 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.946395 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.946642 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:08.947112 (+ 470us) server_negotiation.cc:299] Negotiation successful
0923 17:33:08.947271 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1333,"thread_start_us":1219,"threads_started":1}
I20260923 17:33:08.947576 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.936380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58510 (local address 127.8.29.193:43357)
0923 17:33:08.937896 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:08.937906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:08.938546 (+ 640us) server_negotiation.cc:407] Connection header received
0923 17:33:08.938654 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:08.938661 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:08.938916 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:08.939117 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:08.939905 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.941541 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.944765 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.946138 (+ 1373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.946271 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.946531 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:08.947060 (+ 529us) server_negotiation.cc:299] Negotiation successful
0923 17:33:08.947237 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1360,"thread_start_us":1235,"threads_started":1}
I20260923 17:33:08.948057 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.935416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57512 (local address 127.8.29.195:35127)
0923 17:33:08.936789 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:08.936795 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:08.937384 (+ 589us) server_negotiation.cc:407] Connection header received
0923 17:33:08.937707 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:08.937716 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:08.937989 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:08.938161 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:08.940168 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.941694 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.944410 (+ 2716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.945785 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.945928 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.946234 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:08.947399 (+ 1165us) server_negotiation.cc:299] Negotiation successful
0923 17:33:08.947564 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1216,"thread_start_us":1078,"threads_started":1}
I20260923 17:33:08.948370 27556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.935838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:43357 (local address 127.0.0.1:58510)
0923 17:33:08.938279 (+ 2441us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:08.938296 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:08.938408 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:08.939090 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:08.939096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:08.939124 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:08.939558 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.939575 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.941683 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.941689 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.944538 (+ 2849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.944553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.946655 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.946660 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.946859 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.946886 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 17:33:08.947918 (+ 1032us) client_negotiation.cc:240] Negotiation successful
0923 17:33:08.948057 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2326,"mutex_wait_us":7,"thread_start_us":2199,"threads_started":1}
I20260923 17:33:08.948448 27558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:08.937735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34039 (local address 127.0.0.1:60510)
0923 17:33:08.940150 (+ 2415us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:08.940165 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:08.940264 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:08.940734 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:08.940738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:08.940761 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:08.941079 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.941089 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.943023 (+ 1934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.943029 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.945004 (+ 1975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:08.945015 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:08.946484 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:08.946491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:08.946740 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:08.946778 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:33:08.946869 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 17:33:08.947004 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2286,"mutex_wait_us":22,"thread_start_us":2134,"threads_started":1}
I20260923 17:33:08.952450 27477 tablet_service.cc:1467] Processing CreateTablet for tablet 371ba8bbe76842959044a6f34a035650 (DEFAULT_TABLE table=test-table [id=1317861f79c04c80b628f0bf4db805dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:08.952451 27347 tablet_service.cc:1467] Processing CreateTablet for tablet 371ba8bbe76842959044a6f34a035650 (DEFAULT_TABLE table=test-table [id=1317861f79c04c80b628f0bf4db805dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:08.954067 27477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371ba8bbe76842959044a6f34a035650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:08.956233 27216 tablet_service.cc:1467] Processing CreateTablet for tablet 371ba8bbe76842959044a6f34a035650 (DEFAULT_TABLE table=test-table [id=1317861f79c04c80b628f0bf4db805dc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:08.956831 27347 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371ba8bbe76842959044a6f34a035650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:08.957616 27216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 371ba8bbe76842959044a6f34a035650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:08.968914 27560 tablet_bootstrap.cc:492] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: Bootstrap starting.
I20260923 17:33:08.970350 27561 tablet_bootstrap.cc:492] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: Bootstrap starting.
I20260923 17:33:08.973124 27560 tablet_bootstrap.cc:654] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:08.974097 27562 tablet_bootstrap.cc:492] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: Bootstrap starting.
I20260923 17:33:08.974350 27561 tablet_bootstrap.cc:654] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:08.974566 27560 log.cc:824] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:08.975867 27561 log.cc:824] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:08.977180 27281 heartbeater.cc:503] Master 127.8.29.254:33951 was elected leader, sending a full tablet report...
I20260923 17:33:08.977653 27562 tablet_bootstrap.cc:654] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:08.978752 27562 log.cc:824] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:08.980749 27560 tablet_bootstrap.cc:492] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: No bootstrap required, opened a new log
I20260923 17:33:08.981166 27560 ts_tablet_manager.cc:1397] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260923 17:33:08.985455 27562 tablet_bootstrap.cc:492] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: No bootstrap required, opened a new log
I20260923 17:33:08.985455 27561 tablet_bootstrap.cc:492] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: No bootstrap required, opened a new log
I20260923 17:33:08.985764 27562 ts_tablet_manager.cc:1397] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260923 17:33:08.985771 27561 ts_tablet_manager.cc:1397] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.007s
I20260923 17:33:08.989701 27560 raft_consensus.cc:348] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } }
I20260923 17:33:08.990255 27560 raft_consensus.cc:374] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:08.990389 27560 raft_consensus.cc:729] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: baf5daa3e3a747bab63cb3b2e832c273, State: Initialized, Role: FOLLOWER
I20260923 17:33:08.991074 27560 consensus_queue.cc:260] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } }
I20260923 17:33:08.992174 27412 heartbeater.cc:503] Master 127.8.29.254:33951 was elected leader, sending a full tablet report...
I20260923 17:33:08.992906 27560 ts_tablet_manager.cc:1428] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: Time spent starting tablet: real 0.012s user 0.005s sys 0.005s
I20260923 17:33:08.997006 27561 raft_consensus.cc:348] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } }
I20260923 17:33:08.997556 27561 raft_consensus.cc:374] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:08.997299 27562 raft_consensus.cc:348] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } }
I20260923 17:33:08.997661 27561 raft_consensus.cc:729] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6da1a68f2ee64b25ab4cf2f186ca4589, State: Initialized, Role: FOLLOWER
I20260923 17:33:08.997725 27562 raft_consensus.cc:374] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:08.997822 27562 raft_consensus.cc:729] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 176380baba1e42839c4d1112241e3c65, State: Initialized, Role: FOLLOWER
I20260923 17:33:08.998373 27562 consensus_queue.cc:260] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } }
I20260923 17:33:08.998373 27561 consensus_queue.cc:260] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } }
I20260923 17:33:09.000034 27542 heartbeater.cc:503] Master 127.8.29.254:33951 was elected leader, sending a full tablet report...
I20260923 17:33:09.000284 27562 ts_tablet_manager.cc:1428] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260923 17:33:09.000555 27561 ts_tablet_manager.cc:1428] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: Time spent starting tablet: real 0.015s user 0.008s sys 0.008s
W20260923 17:33:09.128186 27543 tablet.cc:2369] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:09.170408 27568 raft_consensus.cc:482] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:33:09.170732 27568 raft_consensus.cc:504] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } }
W20260923 17:33:09.172101 27413 tablet.cc:2369] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:09.172658 27568 leader_election.cc:288] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6da1a68f2ee64b25ab4cf2f186ca4589 (127.8.29.195:35127), baf5daa3e3a747bab63cb3b2e832c273 (127.8.29.194:34039)
I20260923 17:33:09.179356 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:09.173153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:57605 (local address 127.8.29.195:35127)
0923 17:33:09.173391 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:09.173398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:09.173598 (+ 200us) server_negotiation.cc:407] Connection header received
0923 17:33:09.174017 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:09.174022 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:09.174086 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:09.174160 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:09.174786 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.175482 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.177552 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.178501 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.178653 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:09.178725 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:09.178985 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 17:33:09.179108 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260923 17:33:09.179447 27559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:09.173685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:36067 (local address 127.8.29.194:34039)
0923 17:33:09.173882 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:09.173886 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:09.174232 (+ 346us) server_negotiation.cc:407] Connection header received
0923 17:33:09.174403 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:09.174407 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:09.174509 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:09.174584 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:09.175078 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.175761 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.177732 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.178586 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.178722 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:09.178789 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:09.179079 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 17:33:09.179204 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:33:09.179486 27569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:09.172873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:35127 (local address 127.8.29.193:57605)
0923 17:33:09.173455 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:09.173474 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:09.173649 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:09.174207 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:09.174212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:09.174300 (+ 88us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:09.174651 (+ 351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:09.174661 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.175605 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.175610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:09.177368 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:09.177380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.178668 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.178674 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:09.178837 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:09.178862 (+ 25us) client_negotiation.cc:769] Sending connection context
0923 17:33:09.178947 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 17:33:09.179083 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"mutex_wait_us":36,"spinlock_wait_cycles":44288,"thread_start_us":281,"threads_started":1}
I20260923 17:33:09.179530 27570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:09.173555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:34039 (local address 127.8.29.193:36067)
0923 17:33:09.174124 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:09.174138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:09.174278 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:09.174620 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:09.174624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:09.174650 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:09.174939 (+ 289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:09.174947 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.175903 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.175907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:09.177545 (+ 1638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:09.177558 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.178773 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.178780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:09.178926 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:09.178949 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:33:09.179026 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 17:33:09.179150 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"mutex_wait_us":1,"thread_start_us":283,"threads_started":1}
I20260923 17:33:09.180680 27367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba8bbe76842959044a6f34a035650" candidate_uuid: "176380baba1e42839c4d1112241e3c65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf5daa3e3a747bab63cb3b2e832c273" is_pre_election: true
I20260923 17:33:09.180680 27497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba8bbe76842959044a6f34a035650" candidate_uuid: "176380baba1e42839c4d1112241e3c65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" is_pre_election: true
I20260923 17:33:09.181285 27497 raft_consensus.cc:2454] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176380baba1e42839c4d1112241e3c65 in term 0.
I20260923 17:33:09.181300 27367 raft_consensus.cc:2454] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 176380baba1e42839c4d1112241e3c65 in term 0.
I20260923 17:33:09.182041 27172 leader_election.cc:302] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176380baba1e42839c4d1112241e3c65, 6da1a68f2ee64b25ab4cf2f186ca4589; no voters:
I20260923 17:33:09.182474 27568 raft_consensus.cc:2781] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:33:09.182638 27568 raft_consensus.cc:482] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:33:09.182745 27568 raft_consensus.cc:3037] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:09.184885 27568 raft_consensus.cc:504] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } }
I20260923 17:33:09.185472 27568 leader_election.cc:288] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [CANDIDATE]: Term 1 election: Requested vote from peers 6da1a68f2ee64b25ab4cf2f186ca4589 (127.8.29.195:35127), baf5daa3e3a747bab63cb3b2e832c273 (127.8.29.194:34039)
I20260923 17:33:09.186040 27497 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba8bbe76842959044a6f34a035650" candidate_uuid: "176380baba1e42839c4d1112241e3c65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589"
I20260923 17:33:09.186144 27367 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "371ba8bbe76842959044a6f34a035650" candidate_uuid: "176380baba1e42839c4d1112241e3c65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "baf5daa3e3a747bab63cb3b2e832c273"
I20260923 17:33:09.186247 27497 raft_consensus.cc:3037] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:09.186353 27367 raft_consensus.cc:3037] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:09.188565 27367 raft_consensus.cc:2454] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176380baba1e42839c4d1112241e3c65 in term 1.
I20260923 17:33:09.188659 27497 raft_consensus.cc:2454] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 176380baba1e42839c4d1112241e3c65 in term 1.
I20260923 17:33:09.189081 27172 leader_election.cc:302] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 176380baba1e42839c4d1112241e3c65, baf5daa3e3a747bab63cb3b2e832c273; no voters:
I20260923 17:33:09.189456 27568 raft_consensus.cc:2781] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:09.189949 27568 raft_consensus.cc:686] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 1 LEADER]: Becoming Leader. State: Replica: 176380baba1e42839c4d1112241e3c65, State: Running, Role: LEADER
I20260923 17:33:09.190428 27568 consensus_queue.cc:237] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } }
I20260923 17:33:09.195160 27094 catalog_manager.cc:5495] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 176380baba1e42839c4d1112241e3c65 (127.8.29.193). New cstate: current_term: 1 leader_uuid: "176380baba1e42839c4d1112241e3c65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "176380baba1e42839c4d1112241e3c65" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 43357 } health_report { overall_health: HEALTHY } } }
W20260923 17:33:09.204759 27282 tablet.cc:2369] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:09.247977 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:09.242287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58516 (local address 127.8.29.193:43357)
0923 17:33:09.242528 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:09.242533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:09.242607 (+ 74us) server_negotiation.cc:407] Connection header received
0923 17:33:09.242778 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:09.242782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:09.242887 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:09.243017 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:09.243404 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.244110 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.245556 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.245971 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.246072 (+ 101us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:09.246656 (+ 584us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:09.247221 (+ 565us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:09.247372 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:09.247628 (+ 256us) server_negotiation.cc:299] Negotiation successful
0923 17:33:09.247677 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260923 17:33:09.256306 27497 raft_consensus.cc:1261] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [term 1 FOLLOWER]: Refusing update from remote peer 176380baba1e42839c4d1112241e3c65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:09.256294 27367 raft_consensus.cc:1261] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [term 1 FOLLOWER]: Refusing update from remote peer 176380baba1e42839c4d1112241e3c65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:09.257243 27568 consensus_queue.cc:1035] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "baf5daa3e3a747bab63cb3b2e832c273" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 34039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:09.257560 27571 consensus_queue.cc:1035] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6da1a68f2ee64b25ab4cf2f186ca4589" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 35127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:09.268778 27574 mvcc.cc:205] Tried to move back new op lower bound from 7332596896783663104 to 7332596896528494592. Current Snapshot: MvccSnapshot[applied={T|T < 7332596896783663104}]
I20260923 17:33:09.285573 27094 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:48078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:09.285889 27094 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:48078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:09.289871 27094 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 42da56e783b9474fad9246fa3c10b400: Sending DeleteTablet for 3 replicas of tablet 371ba8bbe76842959044a6f34a035650
I20260923 17:33:09.291172 27216 tablet_service.cc:1514] Processing DeleteTablet for tablet 371ba8bbe76842959044a6f34a035650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:09 UTC) from {username='slave'} at 127.0.0.1:58510
I20260923 17:33:09.291486 27477 tablet_service.cc:1514] Processing DeleteTablet for tablet 371ba8bbe76842959044a6f34a035650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:09 UTC) from {username='slave'} at 127.0.0.1:57512
I20260923 17:33:09.291909 27347 tablet_service.cc:1514] Processing DeleteTablet for tablet 371ba8bbe76842959044a6f34a035650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:09 UTC) from {username='slave'} at 127.0.0.1:60510
I20260923 17:33:09.292097 27584 tablet_replica.cc:330] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: stopping tablet replica
I20260923 17:33:09.292549 27584 raft_consensus.cc:2229] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:33:09.293725 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 27154
I20260923 17:33:09.302325 27586 tablet_replica.cc:330] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: stopping tablet replica
I20260923 17:33:09.303122 27586 raft_consensus.cc:2229] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:09.303781 27586 raft_consensus.cc:2258] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:09.304235 27584 raft_consensus.cc:2258] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:09.306752 27585 tablet_replica.cc:330] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: stopping tablet replica
I20260923 17:33:09.306813 27584 ts_tablet_manager.cc:1905] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:09.307188 27585 raft_consensus.cc:2229] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:09.307691 27585 raft_consensus.cc:2258] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:09.309336 27586 ts_tablet_manager.cc:1905] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:09.309798 27585 ts_tablet_manager.cc:1905] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:09.315160 27584 ts_tablet_manager.cc:1918] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:09.315447 27584 log.cc:1196] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/371ba8bbe76842959044a6f34a035650
I20260923 17:33:09.316035 27584 ts_tablet_manager.cc:1939] T 371ba8bbe76842959044a6f34a035650 P 176380baba1e42839c4d1112241e3c65: Deleting consensus metadata
I20260923 17:33:09.316629 27585 ts_tablet_manager.cc:1918] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:09.316891 27585 log.cc:1196] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/371ba8bbe76842959044a6f34a035650
I20260923 17:33:09.317459 27585 ts_tablet_manager.cc:1939] T 371ba8bbe76842959044a6f34a035650 P 6da1a68f2ee64b25ab4cf2f186ca4589: Deleting consensus metadata
I20260923 17:33:09.318130 27080 catalog_manager.cc:4841] TS 176380baba1e42839c4d1112241e3c65 (127.8.29.193:43357): tablet 371ba8bbe76842959044a6f34a035650 (table test-table [id=1317861f79c04c80b628f0bf4db805dc]) successfully deleted
I20260923 17:33:09.319007 27586 ts_tablet_manager.cc:1918] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:09.319268 27586 log.cc:1196] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/371ba8bbe76842959044a6f34a035650
I20260923 17:33:09.319371 27082 catalog_manager.cc:4841] TS 6da1a68f2ee64b25ab4cf2f186ca4589 (127.8.29.195:35127): tablet 371ba8bbe76842959044a6f34a035650 (table test-table [id=1317861f79c04c80b628f0bf4db805dc]) successfully deleted
I20260923 17:33:09.319866 27586 ts_tablet_manager.cc:1939] T 371ba8bbe76842959044a6f34a035650 P baf5daa3e3a747bab63cb3b2e832c273: Deleting consensus metadata
I20260923 17:33:09.320603 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:09.296287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.8.29.193:43357)
0923 17:33:09.296486 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:09.296493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:09.312210 (+ 15717us) server_negotiation.cc:407] Connection header received
0923 17:33:09.312461 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:09.312468 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:09.312580 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:09.312658 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:09.313340 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.314524 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.316367 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.317005 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.317109 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:09.318781 (+ 1672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:09.318793 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:09.318811 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:09.319661 (+ 850us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:09.319746 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:09.319774 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:09.319898 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:09.320125 (+ 227us) server_negotiation.cc:299] Negotiation successful
0923 17:33:09.320238 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:33:09.321628 27276 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:09.321648 27082 catalog_manager.cc:4841] TS baf5daa3e3a747bab63cb3b2e832c273 (127.8.29.194:34039): tablet 371ba8bbe76842959044a6f34a035650 (table test-table [id=1317861f79c04c80b628f0bf4db805dc]) successfully deleted
I20260923 17:33:09.417896 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 27154
I20260923 17:33:09.445139 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 27285
I20260923 17:33:09.451998 27559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:09.445901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60518 (local address 127.8.29.194:34039)
0923 17:33:09.446097 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:09.446110 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:09.446176 (+ 66us) server_negotiation.cc:407] Connection header received
0923 17:33:09.446392 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:09.446396 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:09.446510 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:09.446603 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:09.447025 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.447770 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.449116 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.449514 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.449623 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:09.450666 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:09.450675 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:09.450690 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:09.451284 (+ 594us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:09.451333 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:09.451354 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:09.451450 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:09.451570 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 17:33:09.451685 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:33:09.452730 27407 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:09.541260 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 27285
I20260923 17:33:09.555713 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 27415
I20260923 17:33:09.562443 27555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:09.556508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57524 (local address 127.8.29.195:35127)
0923 17:33:09.556675 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:09.556681 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:09.556702 (+ 21us) server_negotiation.cc:407] Connection header received
0923 17:33:09.556876 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:09.556881 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:09.556993 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:09.557090 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:09.557516 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.558229 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.559609 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:09.560016 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:09.560115 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:09.561139 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:09.561148 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:09.561163 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:09.561734 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:09.561783 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:09.561804 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:09.561899 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:09.562037 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:33:09.562130 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 17:33:09.563175 27537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:09.653791 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 27415
I20260923 17:33:09.667092 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 27064
I20260923 17:33:09.668200 27125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:09.777134 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 27064
2026-09-23T17:33:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2923 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:09 dist-test-slave-p7jt krb5kdc[27594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:09 dist-test-slave-p7jt krb5kdc[27594](info): set up 2 sockets
Sep 23 17:33:09 dist-test-slave-p7jt krb5kdc[27594](info): commencing operation
krb5kdc: starting...
W20260923 17:33:10.794636 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.979s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:10 dist-test-slave-p7jt krb5kdc[27594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:11.041675 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40183
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:11.370154 27607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:11.370476 27607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:11.370610 27607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:11.377950 27607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:11.378074 27607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:11.378129 27607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:11.378213 27607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:11.378430 27607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:11.389011 27607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:40183
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:11.390373 27607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:11.391748 27607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:11.403425 27612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:11.403370 27613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:11.403570 27615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:11.404265 27607 server_base.cc:1034] running on GCE node
I20260923 17:33:11.405045 27607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:11.406502 27607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:11.407732 27607 hybrid_clock.cc:648] HybridClock initialized: now 1790184791407659 us; error 61 us; skew 500 ppm
I20260923 17:33:11.408095 27607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:11 dist-test-slave-p7jt krb5kdc[27594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184791, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:11.411094 27607 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:11.412599 27607 webserver.cc:460] Webserver started at http://127.8.29.254:43243/ using document root <none> and password file <none>
I20260923 17:33:11.413283 27607 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:11.413399 27607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:11.413697 27607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:11.416695 27607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "b046caea87c74652b2b8167af831786b"
format_stamp: "Formatted at 2026-09-23 17:33:11 on dist-test-slave-p7jt"
server_key: "d482a129032310b4a938a0c620520165"
server_key_iv: "81c6f46b9091ff7f61e2195c10d708a1"
server_key_version: "encryptionkey@0"
I20260923 17:33:11.417445 27607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "b046caea87c74652b2b8167af831786b"
format_stamp: "Formatted at 2026-09-23 17:33:11 on dist-test-slave-p7jt"
server_key: "d482a129032310b4a938a0c620520165"
server_key_iv: "81c6f46b9091ff7f61e2195c10d708a1"
server_key_version: "encryptionkey@0"
I20260923 17:33:11.422529 27607 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260923 17:33:11.425908 27622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:11.427338 27607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:11.427498 27607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "b046caea87c74652b2b8167af831786b"
format_stamp: "Formatted at 2026-09-23 17:33:11 on dist-test-slave-p7jt"
server_key: "d482a129032310b4a938a0c620520165"
server_key_iv: "81c6f46b9091ff7f61e2195c10d708a1"
server_key_version: "encryptionkey@0"
I20260923 17:33:11.427682 27607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:11.447155 27607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:11.451407 27607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:11.451679 27607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:11.469475 27607 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:40183
I20260923 17:33:11.469523 27673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:40183 every 8 connection(s)
I20260923 17:33:11.470858 27607 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:11.474642 27674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:11.475191 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 27607
I20260923 17:33:11.475363 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:11.475778 8311 external_mini_cluster.cc:1428] Setting key fea88b0329093a9e83128aec0a782b4f
I20260923 17:33:11.489218 27674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b: Bootstrap starting.
Sep 23 17:33:11 dist-test-slave-p7jt krb5kdc[27594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:11.492592 27674 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:11.494025 27674 log.cc:824] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:11.497499 27674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b: No bootstrap required, opened a new log
I20260923 17:33:11.500897 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:11.477788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60322 (local address 127.8.29.254:40183)
0923 17:33:11.478669 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:11.478686 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:11.478742 (+ 56us) server_negotiation.cc:407] Connection header received
0923 17:33:11.481264 (+ 2522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:11.481296 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:11.481891 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:11.482404 (+ 513us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:11.483774 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:11.485747 (+ 1973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:11.487157 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:11.488244 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:11.488431 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:11.492421 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:11.492433 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:11.492456 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:11.492517 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:11.496267 (+ 3750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:11.497058 (+ 791us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:11.497067 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:11.497076 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:11.497284 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:11.497777 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:11.497781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:11.497784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:11.498399 (+ 615us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:11.498737 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:11.499333 (+ 596us) server_negotiation.cc:299] Negotiation successful
0923 17:33:11.499792 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":371,"threads_started":1}
I20260923 17:33:11.505656 27674 raft_consensus.cc:348] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046caea87c74652b2b8167af831786b" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40183 } }
I20260923 17:33:11.506034 27674 raft_consensus.cc:374] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:11.506124 27674 raft_consensus.cc:729] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b046caea87c74652b2b8167af831786b, State: Initialized, Role: FOLLOWER
I20260923 17:33:11.506721 27674 consensus_queue.cc:260] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046caea87c74652b2b8167af831786b" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40183 } }
I20260923 17:33:11.506964 27674 raft_consensus.cc:388] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:11.507084 27674 raft_consensus.cc:482] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:11.507225 27674 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:11.509490 27674 raft_consensus.cc:504] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046caea87c74652b2b8167af831786b" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40183 } }
I20260923 17:33:11.510011 27674 leader_election.cc:302] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b046caea87c74652b2b8167af831786b; no voters:
I20260923 17:33:11.510636 27674 leader_election.cc:288] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:11.510737 27679 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:11.511387 27679 raft_consensus.cc:686] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [term 1 LEADER]: Becoming Leader. State: Replica: b046caea87c74652b2b8167af831786b, State: Running, Role: LEADER
I20260923 17:33:11.511894 27679 consensus_queue.cc:237] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046caea87c74652b2b8167af831786b" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40183 } }
I20260923 17:33:11.512610 27674 sys_catalog.cc:564] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:11.514729 27680 sys_catalog.cc:455] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b046caea87c74652b2b8167af831786b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046caea87c74652b2b8167af831786b" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40183 } } }
I20260923 17:33:11.514737 27681 sys_catalog.cc:455] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b046caea87c74652b2b8167af831786b. Latest consensus state: current_term: 1 leader_uuid: "b046caea87c74652b2b8167af831786b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046caea87c74652b2b8167af831786b" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40183 } } }
I20260923 17:33:11.515256 27681 sys_catalog.cc:458] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:11.515147 27680 sys_catalog.cc:458] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:11.517628 27687 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:11.521935 27687 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:11.531018 27687 catalog_manager.cc:1347] Generated new cluster ID: f33c1352ec084d4c8a4a831d3265af64
I20260923 17:33:11.531147 27687 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:11.551879 27687 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:11.553886 27687 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:11.566716 27687 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b: Generated new TSK 0
I20260923 17:33:11.567867 27687 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.29.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
Entry for principal kudu/127.8.29.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab.
I20260923 17:33:11.634747 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40183
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:11.963382 27700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:11.963666 27700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:11.963766 27700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:11.971132 27700 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:11.971269 27700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:11.971390 27700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:33:11.981422 27700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.193.keytab
--principal=kudu/127.8.29.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:11.982735 27700 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:11.984042 27700 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:11.996950 27708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:11.997025 27705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:11.997062 27706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:11.997893 27700 server_base.cc:1034] running on GCE node
I20260923 17:33:11.998553 27700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:11.999549 27700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:12.000816 27700 hybrid_clock.cc:648] HybridClock initialized: now 1790184792000750 us; error 61 us; skew 500 ppm
I20260923 17:33:12.001140 27700 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:12 dist-test-slave-p7jt krb5kdc[27594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:12.004355 27700 init.cc:377] Logged in from keytab as kudu/127.8.29.193@KRBTEST.COM (short username kudu)
I20260923 17:33:12.005760 27700 webserver.cc:460] Webserver started at http://127.8.29.193:39451/ using document root <none> and password file <none>
I20260923 17:33:12.006441 27700 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:12.006589 27700 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:12.006873 27700 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:12.009804 27700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "e5659b2e78e64a338356f307f2558f45"
format_stamp: "Formatted at 2026-09-23 17:33:12 on dist-test-slave-p7jt"
server_key: "55e87e66bb6b8ef9275f5ebb8d67022c"
server_key_iv: "e206834994fb8d3710986bba9774bc18"
server_key_version: "encryptionkey@0"
I20260923 17:33:12.010591 27700 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "e5659b2e78e64a338356f307f2558f45"
format_stamp: "Formatted at 2026-09-23 17:33:12 on dist-test-slave-p7jt"
server_key: "55e87e66bb6b8ef9275f5ebb8d67022c"
server_key_iv: "e206834994fb8d3710986bba9774bc18"
server_key_version: "encryptionkey@0"
I20260923 17:33:12.016228 27700 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:33:12.019968 27715 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:12.021590 27700 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:12.021767 27700 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "e5659b2e78e64a338356f307f2558f45"
format_stamp: "Formatted at 2026-09-23 17:33:12 on dist-test-slave-p7jt"
server_key: "55e87e66bb6b8ef9275f5ebb8d67022c"
server_key_iv: "e206834994fb8d3710986bba9774bc18"
server_key_version: "encryptionkey@0"
I20260923 17:33:12.021992 27700 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:12.045637 27700 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:12.049960 27700 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:12.050334 27700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:12.051599 27700 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:12.053150 27700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:12.053234 27700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:12.053359 27700 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:12.053427 27700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:12.087111 27700 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:46177
I20260923 17:33:12.087172 27827 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:46177 every 8 connection(s)
I20260923 17:33:12.088502 27700 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:33:12.098881 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 27700
I20260923 17:33:12.099050 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:33:12.099510 8311 external_mini_cluster.cc:1428] Setting key 7fc2544c9141a4d30d757491a74d2806
Sep 23 17:33:12 dist-test-slave-p7jt krb5kdc[27594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.193@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:12.107882 27830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:12.091116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:34473 (local address 127.8.29.254:40183)
0923 17:33:12.091559 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:12.091564 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:12.092492 (+ 928us) server_negotiation.cc:407] Connection header received
0923 17:33:12.094334 (+ 1842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:12.094340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:12.094423 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:12.094562 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:12.096135 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:12.096870 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:12.098429 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:12.098879 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:12.098997 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:12.102426 (+ 3429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:12.102431 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:12.102435 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:12.102500 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:12.104871 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:12.105588 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:12.105593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:12.105597 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:12.105720 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:12.106135 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:12.106141 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:12.106145 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:12.106414 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:12.106571 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:12.107287 (+ 716us) server_negotiation.cc:299] Negotiation successful
0923 17:33:12.107519 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":231,"threads_started":1}
I20260923 17:33:12.108880 27831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:12.091613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:40183 (local address 127.8.29.193:34473)
0923 17:33:12.092326 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:12.092370 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:12.094055 (+ 1685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:12.094811 (+ 756us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:12.094820 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:12.095371 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:12.095955 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:12.095973 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:12.097032 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:12.097035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:12.098284 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:12.098297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:12.099114 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:12.099121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:12.099359 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:12.099993 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:12.100023 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:12.102200 (+ 2177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:12.105005 (+ 2805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:12.105015 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:12.105033 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:12.105472 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:12.105832 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:12.105837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:12.105840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:12.106028 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:12.106568 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:12.106576 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:12.106919 (+ 343us) client_negotiation.cc:769] Sending connection context
0923 17:33:12.107166 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 17:33:12.107725 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":275,"threads_started":1}
I20260923 17:33:12.110519 27828 heartbeater.cc:347] Connected to a master server at 127.8.29.254:40183
I20260923 17:33:12.110852 27828 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:12.111545 27828 heartbeater.cc:511] Master 127.8.29.254:40183 requested a full tablet report, sending...
I20260923 17:33:12.113821 27639 ts_manager.cc:194] Registered new tserver with Master: e5659b2e78e64a338356f307f2558f45 (127.8.29.193:46177)
I20260923 17:33:12.115290 27639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.193@KRBTEST.COM'} at 127.8.29.193:34473
WARNING: no policy specified for kudu/127.8.29.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
Entry for principal kudu/127.8.29.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab.
I20260923 17:33:12.154021 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40183
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:12.478994 27834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:12.479266 27834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:12.479387 27834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:12.486896 27834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:12.487013 27834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:12.487165 27834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:33:12.497217 27834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.194.keytab
--principal=kudu/127.8.29.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:12.498556 27834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:12.499943 27834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:12.513144 27840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:12.513216 27839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:12.513136 27842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:12.514245 27834 server_base.cc:1034] running on GCE node
I20260923 17:33:12.514935 27834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:12.515899 27834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:12.517102 27834 hybrid_clock.cc:648] HybridClock initialized: now 1790184792517042 us; error 59 us; skew 500 ppm
I20260923 17:33:12.517460 27834 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:12 dist-test-slave-p7jt krb5kdc[27594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:12.533000 27834 init.cc:377] Logged in from keytab as kudu/127.8.29.194@KRBTEST.COM (short username kudu)
I20260923 17:33:12.534612 27834 webserver.cc:460] Webserver started at http://127.8.29.194:44277/ using document root <none> and password file <none>
I20260923 17:33:12.535358 27834 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:12.535480 27834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:12.535815 27834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:12.538986 27834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "932348919ed8481093f0e2dd0afdbf77"
format_stamp: "Formatted at 2026-09-23 17:33:12 on dist-test-slave-p7jt"
server_key: "4d13cdf257b39253007f6fd4b41384c2"
server_key_iv: "15c52e0bbd593cac33a6335d131a232d"
server_key_version: "encryptionkey@0"
I20260923 17:33:12.539826 27834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "932348919ed8481093f0e2dd0afdbf77"
format_stamp: "Formatted at 2026-09-23 17:33:12 on dist-test-slave-p7jt"
server_key: "4d13cdf257b39253007f6fd4b41384c2"
server_key_iv: "15c52e0bbd593cac33a6335d131a232d"
server_key_version: "encryptionkey@0"
I20260923 17:33:12.545288 27834 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260923 17:33:12.549100 27849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:12.550719 27834 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260923 17:33:12.550877 27834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "932348919ed8481093f0e2dd0afdbf77"
format_stamp: "Formatted at 2026-09-23 17:33:12 on dist-test-slave-p7jt"
server_key: "4d13cdf257b39253007f6fd4b41384c2"
server_key_iv: "15c52e0bbd593cac33a6335d131a232d"
server_key_version: "encryptionkey@0"
I20260923 17:33:12.551074 27834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:12.567087 27834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:12.571640 27834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:12.571933 27834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:12.572942 27834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:12.574343 27834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:12.574417 27834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:12.574579 27834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:12.574635 27834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:12.604466 27834 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:39869
I20260923 17:33:12.604513 27961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:39869 every 8 connection(s)
I20260923 17:33:12.605737 27834 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:33:12.608304 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 27834
I20260923 17:33:12.608500 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:33:12.609059 8311 external_mini_cluster.cc:1428] Setting key 6739e7d87d99b8792a5545fe9e39aee8
Sep 23 17:33:12 dist-test-slave-p7jt krb5kdc[27594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184792, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.194@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:12.625633 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:12.608337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:50979 (local address 127.8.29.254:40183)
0923 17:33:12.608892 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:12.608899 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:12.610526 (+ 1627us) server_negotiation.cc:407] Connection header received
0923 17:33:12.611690 (+ 1164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:12.611695 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:12.611772 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:12.611863 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:12.613468 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:12.614282 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:12.615836 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:12.616274 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:12.616386 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:12.619966 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:12.619971 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:12.619975 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:12.620009 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:12.622487 (+ 2478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:12.623351 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:12.623355 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:12.623358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:12.623503 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:12.623929 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:12.623932 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:12.623934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:12.624198 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:12.624321 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:12.625145 (+ 824us) server_negotiation.cc:299] Negotiation successful
0923 17:33:12.625356 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":282,"threads_started":1}
I20260923 17:33:12.626966 27965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:12.608893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:40183 (local address 127.8.29.194:50979)
0923 17:33:12.609659 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:12.609703 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:12.611432 (+ 1729us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:12.612111 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:12.612119 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:12.612675 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:12.613247 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:12.613273 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:12.614434 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:12.614438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:12.615698 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:12.615710 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:12.616524 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:12.616531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:12.616788 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:12.617464 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:12.617497 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:12.619741 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:12.622639 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:12.622648 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:12.622665 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:12.623246 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:12.623592 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:12.623596 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:12.623599 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:12.623837 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:12.624309 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:12.624317 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:12.624756 (+ 439us) client_negotiation.cc:769] Sending connection context
0923 17:33:12.625045 (+ 289us) client_negotiation.cc:240] Negotiation successful
0923 17:33:12.625589 (+ 544us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":314,"threads_started":1}
I20260923 17:33:12.628502 27962 heartbeater.cc:347] Connected to a master server at 127.8.29.254:40183
I20260923 17:33:12.628809 27962 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:12.629575 27962 heartbeater.cc:511] Master 127.8.29.254:40183 requested a full tablet report, sending...
I20260923 17:33:12.631160 27639 ts_manager.cc:194] Registered new tserver with Master: 932348919ed8481093f0e2dd0afdbf77 (127.8.29.194:39869)
I20260923 17:33:12.632076 27639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.194@KRBTEST.COM'} at 127.8.29.194:50979
WARNING: no policy specified for kudu/127.8.29.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
Entry for principal kudu/127.8.29.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab.
I20260923 17:33:12.666117 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40183
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:42237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:12.986780 27968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:12.987092 27968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:12.987195 27968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:12.994395 27968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:12.994540 27968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:12.994683 27968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:33:13.004516 27968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:42237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.195.keytab
--principal=kudu/127.8.29.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:40183
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:13.005770 27968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:13.007052 27968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:13.020053 27973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:13.020033 27976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:13.019989 27974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:13.020866 27968 server_base.cc:1034] running on GCE node
I20260923 17:33:13.021435 27968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:13.022490 27968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:13.023703 27968 hybrid_clock.cc:648] HybridClock initialized: now 1790184793023644 us; error 54 us; skew 500 ppm
I20260923 17:33:13.024035 27968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:13 dist-test-slave-p7jt krb5kdc[27594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184793, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:13.027344 27968 init.cc:377] Logged in from keytab as kudu/127.8.29.195@KRBTEST.COM (short username kudu)
I20260923 17:33:13.028898 27968 webserver.cc:460] Webserver started at http://127.8.29.195:32907/ using document root <none> and password file <none>
I20260923 17:33:13.029620 27968 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:13.029733 27968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:13.030033 27968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:13.033213 27968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "85b9f7597a344643b51a55baa4cb8e01"
format_stamp: "Formatted at 2026-09-23 17:33:13 on dist-test-slave-p7jt"
server_key: "ad6c360d2b7a5a920467f0e8432af65f"
server_key_iv: "8c4d1de2ea78e14492c953b5737bfb95"
server_key_version: "encryptionkey@0"
I20260923 17:33:13.034021 27968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "85b9f7597a344643b51a55baa4cb8e01"
format_stamp: "Formatted at 2026-09-23 17:33:13 on dist-test-slave-p7jt"
server_key: "ad6c360d2b7a5a920467f0e8432af65f"
server_key_iv: "8c4d1de2ea78e14492c953b5737bfb95"
server_key_version: "encryptionkey@0"
I20260923 17:33:13.039532 27968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20260923 17:33:13.043207 27983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:13.044682 27968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 17:33:13.044862 27968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "85b9f7597a344643b51a55baa4cb8e01"
format_stamp: "Formatted at 2026-09-23 17:33:13 on dist-test-slave-p7jt"
server_key: "ad6c360d2b7a5a920467f0e8432af65f"
server_key_iv: "8c4d1de2ea78e14492c953b5737bfb95"
server_key_version: "encryptionkey@0"
I20260923 17:33:13.045074 27968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:13.062839 27968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:13.067245 27968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:13.067523 27968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:13.068565 27968 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:13.069964 27968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:13.070047 27968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:13.070183 27968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:13.070231 27968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:13.114598 27968 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:40841
I20260923 17:33:13.114676 28095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:40841 every 8 connection(s)
I20260923 17:33:13.116374 27968 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:33:13.118508 27828 heartbeater.cc:503] Master 127.8.29.254:40183 was elected leader, sending a full tablet report...
I20260923 17:33:13.120253 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 27968
I20260923 17:33:13.120433 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:33:13.121074 8311 external_mini_cluster.cc:1428] Setting key 87461c27015070b82e4ddac26900dc75
Sep 23 17:33:13 dist-test-slave-p7jt krb5kdc[27594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184793, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.195@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:13.139389 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.119668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:55129 (local address 127.8.29.254:40183)
0923 17:33:13.121369 (+ 1701us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.121377 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.121403 (+ 26us) server_negotiation.cc:407] Connection header received
0923 17:33:13.123939 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.123946 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.124039 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.124149 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:13.126045 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.127087 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.128887 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.129412 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.129543 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.133447 (+ 3904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:13.133453 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:13.133457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:13.133498 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:13.136363 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.137178 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.137183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.137185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.137316 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.137773 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.137777 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.137779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.138090 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:13.138211 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.138978 (+ 767us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.139156 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1560}
I20260923 17:33:13.140379 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.120138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:40183 (local address 127.8.29.195:55129)
0923 17:33:13.121115 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.121243 (+ 128us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:13.123612 (+ 2369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:13.124414 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:13.124423 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:13.125185 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:13.125841 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.125870 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.127254 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.127261 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.128729 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.128744 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.129671 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.129680 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.129946 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.130706 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:13.130739 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:13.133183 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:13.136506 (+ 3323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:13.136516 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:13.136535 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:13.137062 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:13.137439 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:13.137443 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:13.137447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:13.137689 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:13.138215 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:13.138221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:13.138642 (+ 421us) client_negotiation.cc:769] Sending connection context
0923 17:33:13.138889 (+ 247us) client_negotiation.cc:240] Negotiation successful
0923 17:33:13.139316 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":442,"threads_started":1}
I20260923 17:33:13.141767 28096 heartbeater.cc:347] Connected to a master server at 127.8.29.254:40183
I20260923 17:33:13.142077 28096 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:13.142980 28096 heartbeater.cc:511] Master 127.8.29.254:40183 requested a full tablet report, sending...
I20260923 17:33:13.144582 27639 ts_manager.cc:194] Registered new tserver with Master: 85b9f7597a344643b51a55baa4cb8e01 (127.8.29.195:40841)
I20260923 17:33:13.145426 27639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.29.195@KRBTEST.COM'} at 127.8.29.195:55129
I20260923 17:33:13.150290 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:33:13.166399 27964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.154582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60332 (local address 127.8.29.254:40183)
0923 17:33:13.154817 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.154823 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.155109 (+ 286us) server_negotiation.cc:407] Connection header received
0923 17:33:13.155291 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.155295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.155379 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.155459 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:13.156866 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.157611 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.158922 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.159336 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.159429 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.161679 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:13.161684 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:13.161687 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:13.161730 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:13.164091 (+ 2361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.164677 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.164680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.164682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.164795 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.165116 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.165119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.165121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.165448 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:13.165655 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.165919 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.166086 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260923 17:33:13.170202 27639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60332:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:33:13.177067 27639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:33:13.201027 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.190434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53516 (local address 127.8.29.194:39869)
0923 17:33:13.192085 (+ 1651us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.192095 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.192573 (+ 478us) server_negotiation.cc:407] Connection header received
0923 17:33:13.192897 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.192905 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.193184 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.193366 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:13.194264 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.195738 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.198356 (+ 2618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.199746 (+ 1390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.199893 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.200096 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.200493 (+ 397us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.200665 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1507,"thread_start_us":1378,"threads_started":1}
I20260923 17:33:13.201234 28110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.189724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.0.0.1:43828)
0923 17:33:13.191316 (+ 1592us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.191365 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:13.191553 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:13.194387 (+ 2834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:13.194391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:13.194417 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:13.194797 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.194812 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.196340 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.196345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.198652 (+ 2307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.198667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.200383 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.200390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.200572 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.200604 (+ 32us) client_negotiation.cc:769] Sending connection context
0923 17:33:13.200673 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 17:33:13.200805 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1290,"mutex_wait_us":29,"thread_start_us":2436,"threads_started":1}
I20260923 17:33:13.201840 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.190206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:39869 (local address 127.0.0.1:53516)
0923 17:33:13.192452 (+ 2246us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.192469 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:13.192683 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:13.193671 (+ 988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:13.193679 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:13.193709 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:13.194065 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.194077 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.195876 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.195882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.198205 (+ 2323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.198222 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.199973 (+ 1751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.199979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.200190 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.200263 (+ 73us) client_negotiation.cc:769] Sending connection context
0923 17:33:13.201315 (+ 1052us) client_negotiation.cc:240] Negotiation successful
0923 17:33:13.201456 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2095,"mutex_wait_us":81,"thread_start_us":623,"threads_started":1}
I20260923 17:33:13.202096 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.192902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54370 (local address 127.8.29.195:40841)
0923 17:33:13.194149 (+ 1247us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.194155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.194182 (+ 27us) server_negotiation.cc:407] Connection header received
0923 17:33:13.194274 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.194280 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.194552 (+ 272us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.194763 (+ 211us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:13.195299 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.196582 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.199037 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.199984 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.200921 (+ 937us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.201138 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.201595 (+ 457us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.201762 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1097,"thread_start_us":947,"threads_started":1}
I20260923 17:33:13.202411 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.189955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43828 (local address 127.8.29.193:46177)
0923 17:33:13.192603 (+ 2648us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.192612 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.192635 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:33:13.193902 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.193919 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.194195 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.194385 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:13.194999 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.196202 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.198816 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.200181 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.200293 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.200511 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.201930 (+ 1419us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.202109 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2488,"thread_start_us":2312,"threads_started":1}
I20260923 17:33:13.202598 28112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.192549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:40841 (local address 127.0.0.1:54370)
0923 17:33:13.193064 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.193081 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:13.193201 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:13.194738 (+ 1537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:13.194746 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:13.194773 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:13.195120 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.195132 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.196733 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.196750 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.198899 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.198909 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.200195 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.200201 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.200383 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.200413 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 17:33:13.200472 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 17:33:13.200664 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":60,"thread_start_us":611,"threads_started":1}
I20260923 17:33:13.205722 28031 tablet_service.cc:1467] Processing CreateTablet for tablet fc6507f2fb614376bfbcd4dbc8d7fbee (DEFAULT_TABLE table=test-table [id=531873e5c5e84f3bad19048c04dce3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:13.205657 27897 tablet_service.cc:1467] Processing CreateTablet for tablet fc6507f2fb614376bfbcd4dbc8d7fbee (DEFAULT_TABLE table=test-table [id=531873e5c5e84f3bad19048c04dce3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:13.207193 27897 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6507f2fb614376bfbcd4dbc8d7fbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:13.207209 28031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6507f2fb614376bfbcd4dbc8d7fbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:13.208606 27763 tablet_service.cc:1467] Processing CreateTablet for tablet fc6507f2fb614376bfbcd4dbc8d7fbee (DEFAULT_TABLE table=test-table [id=531873e5c5e84f3bad19048c04dce3dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:13.210036 27763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc6507f2fb614376bfbcd4dbc8d7fbee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:13.219496 28114 tablet_bootstrap.cc:492] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: Bootstrap starting.
I20260923 17:33:13.223264 28114 tablet_bootstrap.cc:654] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:13.223320 28115 tablet_bootstrap.cc:492] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: Bootstrap starting.
I20260923 17:33:13.224795 28116 tablet_bootstrap.cc:492] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45: Bootstrap starting.
I20260923 17:33:13.224951 28114 log.cc:824] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:13.227279 28115 tablet_bootstrap.cc:654] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:13.227895 28116 tablet_bootstrap.cc:654] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:13.228492 28114 tablet_bootstrap.cc:492] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: No bootstrap required, opened a new log
I20260923 17:33:13.228529 28115 log.cc:824] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:13.228778 28114 ts_tablet_manager.cc:1397] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260923 17:33:13.228859 28116 log.cc:824] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:13.233069 28115 tablet_bootstrap.cc:492] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: No bootstrap required, opened a new log
I20260923 17:33:13.233350 28115 ts_tablet_manager.cc:1397] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260923 17:33:13.233408 28116 tablet_bootstrap.cc:492] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45: No bootstrap required, opened a new log
I20260923 17:33:13.233702 28116 ts_tablet_manager.cc:1397] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260923 17:33:13.239549 28114 raft_consensus.cc:348] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.240064 28114 raft_consensus.cc:374] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:13.240180 28114 raft_consensus.cc:729] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85b9f7597a344643b51a55baa4cb8e01, State: Initialized, Role: FOLLOWER
I20260923 17:33:13.241010 28114 consensus_queue.cc:260] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.241359 28115 raft_consensus.cc:348] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.241966 28115 raft_consensus.cc:374] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:13.242070 28115 raft_consensus.cc:729] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932348919ed8481093f0e2dd0afdbf77, State: Initialized, Role: FOLLOWER
I20260923 17:33:13.243039 28096 heartbeater.cc:503] Master 127.8.29.254:40183 was elected leader, sending a full tablet report...
I20260923 17:33:13.242890 28115 consensus_queue.cc:260] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.244181 28116 raft_consensus.cc:348] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.244674 28116 raft_consensus.cc:374] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:13.244764 28116 raft_consensus.cc:729] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5659b2e78e64a338356f307f2558f45, State: Initialized, Role: FOLLOWER
I20260923 17:33:13.245527 28116 consensus_queue.cc:260] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.248950 28120 raft_consensus.cc:482] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:33:13.249346 28120 raft_consensus.cc:504] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.251021 28114 ts_tablet_manager.cc:1428] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: Time spent starting tablet: real 0.022s user 0.017s sys 0.000s
I20260923 17:33:13.251983 28120 leader_election.cc:288] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5659b2e78e64a338356f307f2558f45 (127.8.29.193:46177), 932348919ed8481093f0e2dd0afdbf77 (127.8.29.194:39869)
I20260923 17:33:13.254443 28116 ts_tablet_manager.cc:1428] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45: Time spent starting tablet: real 0.021s user 0.009s sys 0.011s
I20260923 17:33:13.260129 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.252232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.8.29.195:60625)
0923 17:33:13.252524 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.252548 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:13.252682 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:13.253431 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:13.253436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:13.253465 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:13.253653 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.253665 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.255146 (+ 1481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.255151 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.257285 (+ 2134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.257305 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.259285 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.259297 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.259567 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.259606 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 17:33:13.259701 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:33:13.259832 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260923 17:33:13.260700 27962 heartbeater.cc:503] Master 127.8.29.254:40183 was elected leader, sending a full tablet report...
I20260923 17:33:13.260774 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.252722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:60625 (local address 127.8.29.193:46177)
0923 17:33:13.252944 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.252949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.252971 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:33:13.253053 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.253057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.253136 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.253224 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:13.253842 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.254973 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.257515 (+ 2542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.258898 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.259063 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.259147 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.260420 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.260555 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 17:33:13.261636 28115 ts_tablet_manager.cc:1428] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: Time spent starting tablet: real 0.028s user 0.021s sys 0.005s
I20260923 17:33:13.262697 27783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6507f2fb614376bfbcd4dbc8d7fbee" candidate_uuid: "85b9f7597a344643b51a55baa4cb8e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5659b2e78e64a338356f307f2558f45" is_pre_election: true
I20260923 17:33:13.262993 28122 raft_consensus.cc:482] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:33:13.263284 27783 raft_consensus.cc:2454] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b9f7597a344643b51a55baa4cb8e01 in term 0.
I20260923 17:33:13.263280 28122 raft_consensus.cc:504] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.265527 28122 leader_election.cc:288] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5659b2e78e64a338356f307f2558f45 (127.8.29.193:46177), 85b9f7597a344643b51a55baa4cb8e01 (127.8.29.195:40841)
I20260923 17:33:13.265538 27986 leader_election.cc:302] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b9f7597a344643b51a55baa4cb8e01, e5659b2e78e64a338356f307f2558f45; no voters:
I20260923 17:33:13.266518 28121 raft_consensus.cc:2781] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:33:13.266690 28121 raft_consensus.cc:482] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:33:13.266831 28121 raft_consensus.cc:3037] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:13.269271 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.259913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:48635 (local address 127.8.29.194:39869)
0923 17:33:13.260109 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.260114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.261549 (+ 1435us) server_negotiation.cc:407] Connection header received
0923 17:33:13.261819 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.261824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.261896 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.261956 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:13.262613 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.263836 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.266366 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.267762 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.267916 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.268001 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.268634 (+ 633us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.268789 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:33:13.270062 28098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.260013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:39869 (local address 127.8.29.195:48635)
0923 17:33:13.261460 (+ 1447us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.261481 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:13.261657 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:13.262159 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:13.262163 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:13.262197 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:13.262409 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.262422 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.264036 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.264042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.266139 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.266160 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.268172 (+ 2012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.268179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.268431 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.268469 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 17:33:13.269486 (+ 1017us) client_negotiation.cc:240] Negotiation successful
0923 17:33:13.269689 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1342,"thread_start_us":2101,"threads_started":1}
I20260923 17:33:13.274523 28121 raft_consensus.cc:504] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.276005 27783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6507f2fb614376bfbcd4dbc8d7fbee" candidate_uuid: "85b9f7597a344643b51a55baa4cb8e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5659b2e78e64a338356f307f2558f45"
I20260923 17:33:13.276293 27783 raft_consensus.cc:3037] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:13.279927 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.266185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:44461 (local address 127.8.29.193:46177)
0923 17:33:13.266358 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.266363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.271097 (+ 4734us) server_negotiation.cc:407] Connection header received
0923 17:33:13.271425 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.271430 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.271521 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.271592 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:13.272516 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.273786 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.276873 (+ 3087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.278516 (+ 1643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.278692 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.278778 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.279471 (+ 693us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.279623 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":61}
I20260923 17:33:13.280000 27783 raft_consensus.cc:2454] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b9f7597a344643b51a55baa4cb8e01 in term 1.
I20260923 17:33:13.279981 27917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6507f2fb614376bfbcd4dbc8d7fbee" candidate_uuid: "85b9f7597a344643b51a55baa4cb8e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932348919ed8481093f0e2dd0afdbf77" is_pre_election: true
I20260923 17:33:13.280486 27986 leader_election.cc:302] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 85b9f7597a344643b51a55baa4cb8e01, e5659b2e78e64a338356f307f2558f45; no voters:
I20260923 17:33:13.280512 27917 raft_consensus.cc:2454] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 85b9f7597a344643b51a55baa4cb8e01 in term 0.
I20260923 17:33:13.280807 28126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.266054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.8.29.194:44461)
0923 17:33:13.270979 (+ 4925us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.271000 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:13.271227 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:13.271809 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:13.271815 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:13.271844 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:13.272290 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.272302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.274028 (+ 1726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.274035 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.276617 (+ 2582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.276641 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.279022 (+ 2381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.279031 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.279242 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.279276 (+ 34us) client_negotiation.cc:769] Sending connection context
0923 17:33:13.280186 (+ 910us) client_negotiation.cc:240] Negotiation successful
0923 17:33:13.280351 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4775,"thread_start_us":4560,"threads_started":1}
I20260923 17:33:13.280824 28120 raft_consensus.cc:2781] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:13.281831 28120 raft_consensus.cc:686] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 1 LEADER]: Becoming Leader. State: Replica: 85b9f7597a344643b51a55baa4cb8e01, State: Running, Role: LEADER
I20260923 17:33:13.282270 27782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6507f2fb614376bfbcd4dbc8d7fbee" candidate_uuid: "932348919ed8481093f0e2dd0afdbf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5659b2e78e64a338356f307f2558f45" is_pre_election: true
I20260923 17:33:13.282882 27782 raft_consensus.cc:2379] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 932348919ed8481093f0e2dd0afdbf77 in current term 1: Already voted for candidate 85b9f7597a344643b51a55baa4cb8e01 in this term.
I20260923 17:33:13.282667 28120 consensus_queue.cc:237] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } }
I20260923 17:33:13.282855 27917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6507f2fb614376bfbcd4dbc8d7fbee" candidate_uuid: "85b9f7597a344643b51a55baa4cb8e01" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932348919ed8481093f0e2dd0afdbf77"
I20260923 17:33:13.283056 27917 raft_consensus.cc:3037] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:13.281880 28121 leader_election.cc:288] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [CANDIDATE]: Term 1 election: Requested vote from peers e5659b2e78e64a338356f307f2558f45 (127.8.29.193:46177), 932348919ed8481093f0e2dd0afdbf77 (127.8.29.194:39869)
I20260923 17:33:13.286618 27917 raft_consensus.cc:2454] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 85b9f7597a344643b51a55baa4cb8e01 in term 1.
I20260923 17:33:13.287956 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.271305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:39139 (local address 127.8.29.195:40841)
0923 17:33:13.271514 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.271520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.279473 (+ 7953us) server_negotiation.cc:407] Connection header received
0923 17:33:13.279747 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.279754 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.279845 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.279910 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:13.281337 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.282381 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.285494 (+ 3113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.286874 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.287000 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.287072 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.287595 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.287722 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 17:33:13.288015 28127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.271143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:40841 (local address 127.8.29.194:39139)
0923 17:33:13.279374 (+ 8231us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.279393 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:13.279583 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:13.280359 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:13.280363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:13.280390 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:13.281157 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.281170 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.282892 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.282898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.285278 (+ 2380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:13.285300 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.287309 (+ 2009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.287313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:13.287438 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.287461 (+ 23us) client_negotiation.cc:769] Sending connection context
0923 17:33:13.287545 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 17:33:13.287685 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8084,"thread_start_us":7885,"threads_started":1}
I20260923 17:33:13.288215 27639 catalog_manager.cc:5495] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 reported cstate change: term changed from 0 to 1, leader changed from <none> to 85b9f7597a344643b51a55baa4cb8e01 (127.8.29.195). New cstate: current_term: 1 leader_uuid: "85b9f7597a344643b51a55baa4cb8e01" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85b9f7597a344643b51a55baa4cb8e01" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 40841 } health_report { overall_health: HEALTHY } } }
I20260923 17:33:13.288807 28051 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc6507f2fb614376bfbcd4dbc8d7fbee" candidate_uuid: "932348919ed8481093f0e2dd0afdbf77" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85b9f7597a344643b51a55baa4cb8e01" is_pre_election: true
I20260923 17:33:13.289745 27852 leader_election.cc:302] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 932348919ed8481093f0e2dd0afdbf77; no voters: 85b9f7597a344643b51a55baa4cb8e01, e5659b2e78e64a338356f307f2558f45
I20260923 17:33:13.290206 28125 raft_consensus.cc:2726] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 17:33:13.303763 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.298238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54384 (local address 127.8.29.195:40841)
0923 17:33:13.298442 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.298449 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.298553 (+ 104us) server_negotiation.cc:407] Connection header received
0923 17:33:13.298808 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.298812 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.298889 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.299023 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:13.299399 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.300131 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.301498 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.301920 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.302027 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.302607 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:13.303134 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:13.303277 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.303387 (+ 110us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.303475 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:33:13.311950 27782 raft_consensus.cc:1261] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 1 FOLLOWER]: Refusing update from remote peer 85b9f7597a344643b51a55baa4cb8e01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:13.311975 27917 raft_consensus.cc:1261] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 1 FOLLOWER]: Refusing update from remote peer 85b9f7597a344643b51a55baa4cb8e01: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:13.312795 28121 consensus_queue.cc:1035] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "932348919ed8481093f0e2dd0afdbf77" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 39869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:13.313097 28120 consensus_queue.cc:1035] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5659b2e78e64a338356f307f2558f45" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 46177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:13.325361 28131 mvcc.cc:205] Tried to move back new op lower bound from 7332596913395523584 to 7332596913291137024. Current Snapshot: MvccSnapshot[applied={T|T < 7332596913395523584}]
I20260923 17:33:13.328022 28133 mvcc.cc:205] Tried to move back new op lower bound from 7332596913395523584 to 7332596913291137024. Current Snapshot: MvccSnapshot[applied={T|T < 7332596913395523584}]
I20260923 17:33:13.329879 28132 mvcc.cc:205] Tried to move back new op lower bound from 7332596913395523584 to 7332596913291137024. Current Snapshot: MvccSnapshot[applied={T|T < 7332596913395523584}]
I20260923 17:33:13.341001 27638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:13.341361 27638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60332:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:13.345453 27638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b046caea87c74652b2b8167af831786b: Sending DeleteTablet for 3 replicas of tablet fc6507f2fb614376bfbcd4dbc8d7fbee
I20260923 17:33:13.346835 27763 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6507f2fb614376bfbcd4dbc8d7fbee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:13 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:43828
I20260923 17:33:13.347093 27897 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6507f2fb614376bfbcd4dbc8d7fbee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:13 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:53516
I20260923 17:33:13.348133 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 27700
I20260923 17:33:13.348079 28031 tablet_service.cc:1514] Processing DeleteTablet for tablet fc6507f2fb614376bfbcd4dbc8d7fbee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:13 UTC) from {username='kudu', principal='kudu/127.8.29.254@KRBTEST.COM'} at 127.0.0.1:54370
Sep 23 17:33:13 dist-test-slave-p7jt krb5kdc[27594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.193@KRBTEST.COM
I20260923 17:33:13.373724 28142 tablet_replica.cc:330] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: stopping tablet replica
I20260923 17:33:13.374256 28142 raft_consensus.cc:2229] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:33:13.375073 28142 raft_consensus.cc:2258] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:13.377393 28142 ts_tablet_manager.cc:1905] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:13.378995 28141 tablet_replica.cc:330] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: stopping tablet replica
I20260923 17:33:13.379483 28141 raft_consensus.cc:2229] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:13.380045 28109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.349487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43830 (local address 127.8.29.193:46177)
0923 17:33:13.349681 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.349687 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.359665 (+ 9978us) server_negotiation.cc:407] Connection header received
0923 17:33:13.360189 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.360196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.360291 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.360364 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:13.362220 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.363422 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.365571 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.366261 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.366362 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.372072 (+ 5710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:13.372096 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:13.372120 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:13.372192 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:13.376083 (+ 3891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.377142 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.377151 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.377155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.377319 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.377878 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.377896 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.377898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.378367 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:13.378673 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.379374 (+ 701us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.379696 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:33:13.380021 28141 raft_consensus.cc:2258] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:13.381610 28140 tablet_replica.cc:330] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45: stopping tablet replica
I20260923 17:33:13.381804 27823 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:13.384389 28141 ts_tablet_manager.cc:1905] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:13.387931 28142 ts_tablet_manager.cc:1918] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:13.388159 28142 log.cc:1196] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/fc6507f2fb614376bfbcd4dbc8d7fbee
I20260923 17:33:13.388799 28142 ts_tablet_manager.cc:1939] T fc6507f2fb614376bfbcd4dbc8d7fbee P 85b9f7597a344643b51a55baa4cb8e01: Deleting consensus metadata
I20260923 17:33:13.390775 27625 catalog_manager.cc:4841] TS 85b9f7597a344643b51a55baa4cb8e01 (127.8.29.195:40841): tablet fc6507f2fb614376bfbcd4dbc8d7fbee (table test-table [id=531873e5c5e84f3bad19048c04dce3dd]) successfully deleted
I20260923 17:33:13.391242 28141 ts_tablet_manager.cc:1918] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:13.391382 28141 log.cc:1196] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/fc6507f2fb614376bfbcd4dbc8d7fbee
I20260923 17:33:13.391790 28141 ts_tablet_manager.cc:1939] T fc6507f2fb614376bfbcd4dbc8d7fbee P 932348919ed8481093f0e2dd0afdbf77: Deleting consensus metadata
I20260923 17:33:13.393280 27626 catalog_manager.cc:4841] TS 932348919ed8481093f0e2dd0afdbf77 (127.8.29.194:39869): tablet fc6507f2fb614376bfbcd4dbc8d7fbee (table test-table [id=531873e5c5e84f3bad19048c04dce3dd]) successfully deleted
I20260923 17:33:13.382079 28140 raft_consensus.cc:2229] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:13.498106 28140 raft_consensus.cc:2258] T fc6507f2fb614376bfbcd4dbc8d7fbee P e5659b2e78e64a338356f307f2558f45 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:13.498175 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 27700
W20260923 17:33:13.512792 27625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.29.193:46177 (error 108)
I20260923 17:33:13.514021 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 27834
I20260923 17:33:13.514766 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.514241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.0.0.1:43840)
0923 17:33:13.514472 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.514593 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 17:33:13.515179 27625 catalog_manager.cc:4584] TS e5659b2e78e64a338356f307f2558f45 (127.8.29.193:46177): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc6507f2fb614376bfbcd4dbc8d7fbee: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46177: connect: Connection refused (error 111)
Sep 23 17:33:13 dist-test-slave-p7jt krb5kdc[27594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.194@KRBTEST.COM
I20260923 17:33:13.531365 28111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.514825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53518 (local address 127.8.29.194:39869)
0923 17:33:13.515004 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.515010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.515030 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:33:13.515195 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.515201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.515299 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.515408 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:13.516741 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.517844 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.519237 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.519886 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.520018 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.524398 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:13.524418 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:13.524440 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:13.524485 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:13.528206 (+ 3721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.529017 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.529032 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.529036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.529197 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.529589 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.529595 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.529598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.530086 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:13.530394 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.530824 (+ 430us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.531051 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260923 17:33:13.532089 27957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:13.568979 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.568419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.0.0.1:43844)
0923 17:33:13.568667 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.568785 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260923 17:33:13.570169 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.569756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.0.0.1:43852)
0923 17:33:13.569918 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.570032 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260923 17:33:13.623078 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 27834
I20260923 17:33:13.630776 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.630298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.0.0.1:43858)
0923 17:33:13.630535 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.630621 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 17:33:13.631822 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.631452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.0.0.1:43864)
0923 17:33:13.631608 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.631683 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 17:33:13.645927 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 27968
Sep 23 17:33:13 dist-test-slave-p7jt krb5kdc[27594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184790, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.195@KRBTEST.COM
I20260923 17:33:13.665704 28113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.646691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54396 (local address 127.8.29.195:40841)
0923 17:33:13.646879 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:13.646884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:13.646935 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:33:13.647200 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:13.647205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:13.647296 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:13.647385 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:13.649174 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.650037 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.652108 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:13.652667 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:13.652817 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:13.658998 (+ 6181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:13.659018 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:13.659039 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:13.659108 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:13.662826 (+ 3718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.663563 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.663571 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.663574 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.663726 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:13.664122 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:13.664126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:13.664128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:13.664544 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:13.664840 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:13.665143 (+ 303us) server_negotiation.cc:299] Negotiation successful
0923 17:33:13.665385 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 17:33:13.666584 28091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:13.712270 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.711821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.0.0.1:43870)
0923 17:33:13.712026 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.712117 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 17:33:13.713460 28108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:13.712977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:46177 (local address 127.0.0.1:43884)
0923 17:33:13.713262 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:13.713342 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:46177: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260923 17:33:13.789467 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 27968
I20260923 17:33:13.805323 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 27607
I20260923 17:33:13.806416 27669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:13.919215 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 27607
2026-09-23T17:33:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4142 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T17:33:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:13Z Disabled control of system clock
I20260923 17:33:13.992594 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:45925
--webserver_interface=127.8.29.254
--webserver_port=0
--builtin_ntp_servers=127.8.29.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:45925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:33:14.321259 28152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:14.321568 28152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:14.321653 28152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:14.329090 28152 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:14.329208 28152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:14.329252 28152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:14.329308 28152 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:14.329350 28152 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:14.338851 28152 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:45925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:45925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:14.340142 28152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:14.341886 28152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:14.353941 28157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:14.353894 28158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:14.353865 28160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:14.354614 28152 server_base.cc:1034] running on GCE node
I20260923 17:33:14.355469 28152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:14.356935 28152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:14.358155 28152 hybrid_clock.cc:648] HybridClock initialized: now 1790184794358069 us; error 64 us; skew 500 ppm
I20260923 17:33:14.358573 28152 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:14.360947 28152 webserver.cc:460] Webserver started at http://127.8.29.254:33493/ using document root <none> and password file <none>
I20260923 17:33:14.361658 28152 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:14.361755 28152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:14.362088 28152 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:14.365186 28152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "f6e4870f0cfb476db5f797c841139943"
format_stamp: "Formatted at 2026-09-23 17:33:14 on dist-test-slave-p7jt"
server_key: "1a6b6f3143da789172e3c90300f7c2da"
server_key_iv: "ee6d4ea41736d737fa3d8bbd03a3a4dc"
server_key_version: "encryptionkey@0"
I20260923 17:33:14.366024 28152 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "f6e4870f0cfb476db5f797c841139943"
format_stamp: "Formatted at 2026-09-23 17:33:14 on dist-test-slave-p7jt"
server_key: "1a6b6f3143da789172e3c90300f7c2da"
server_key_iv: "ee6d4ea41736d737fa3d8bbd03a3a4dc"
server_key_version: "encryptionkey@0"
I20260923 17:33:14.371382 28152 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:33:14.374759 28166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:14.376117 28152 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:33:14.376279 28152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "f6e4870f0cfb476db5f797c841139943"
format_stamp: "Formatted at 2026-09-23 17:33:14 on dist-test-slave-p7jt"
server_key: "1a6b6f3143da789172e3c90300f7c2da"
server_key_iv: "ee6d4ea41736d737fa3d8bbd03a3a4dc"
server_key_version: "encryptionkey@0"
I20260923 17:33:14.376488 28152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:14.401376 28152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:14.402276 28152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:14.402595 28152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:14.420445 28152 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:45925
I20260923 17:33:14.420463 28217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:45925 every 8 connection(s)
I20260923 17:33:14.421814 28152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:14.425616 28218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:14.427210 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 28152
I20260923 17:33:14.427377 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:14.427809 8311 external_mini_cluster.cc:1428] Setting key 3041451b69f052bb58c9e3292adde8f0
I20260923 17:33:14.438263 28218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943: Bootstrap starting.
I20260923 17:33:14.441052 28221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:14.429589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51296 (local address 127.8.29.254:45925)
0923 17:33:14.430728 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:14.430739 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:14.430783 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:33:14.432441 (+ 1658us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:14.432468 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:14.432962 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:14.433331 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:14.433895 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:14.435248 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:14.436577 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:14.437291 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:14.437465 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:14.438660 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:14.438668 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:14.438684 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:14.439245 (+ 561us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:14.439325 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:14.439347 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:14.439448 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:14.439714 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 17:33:14.440037 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":671,"thread_start_us":386,"threads_started":1}
I20260923 17:33:14.441766 28218 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:14.442847 28218 log.cc:824] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:14.445565 28218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943: No bootstrap required, opened a new log
I20260923 17:33:14.452840 28218 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e4870f0cfb476db5f797c841139943" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 45925 } }
I20260923 17:33:14.453267 28218 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:14.453361 28218 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6e4870f0cfb476db5f797c841139943, State: Initialized, Role: FOLLOWER
I20260923 17:33:14.453986 28218 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e4870f0cfb476db5f797c841139943" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 45925 } }
I20260923 17:33:14.454234 28218 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:14.454363 28218 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:14.454546 28218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:14.456812 28218 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e4870f0cfb476db5f797c841139943" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 45925 } }
I20260923 17:33:14.457382 28218 leader_election.cc:302] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6e4870f0cfb476db5f797c841139943; no voters:
I20260923 17:33:14.458067 28218 leader_election.cc:288] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:14.458204 28223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:14.458889 28223 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [term 1 LEADER]: Becoming Leader. State: Replica: f6e4870f0cfb476db5f797c841139943, State: Running, Role: LEADER
I20260923 17:33:14.459445 28223 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e4870f0cfb476db5f797c841139943" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 45925 } }
I20260923 17:33:14.460180 28218 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:14.461937 28224 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6e4870f0cfb476db5f797c841139943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e4870f0cfb476db5f797c841139943" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 45925 } } }
I20260923 17:33:14.462385 28224 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:14.462037 28225 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6e4870f0cfb476db5f797c841139943. Latest consensus state: current_term: 1 leader_uuid: "f6e4870f0cfb476db5f797c841139943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6e4870f0cfb476db5f797c841139943" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 45925 } } }
I20260923 17:33:14.462901 28225 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:14.464859 28230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:14.470268 28230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:14.479620 28230 catalog_manager.cc:1347] Generated new cluster ID: a8953c5279794c238f9fd33d2e0da915
I20260923 17:33:14.479735 28230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:14.489720 28230 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:14.491163 28230 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:14.499943 28230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943: Generated new TSK 0
I20260923 17:33:14.500765 28230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:14.517076 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.193:0
--local_ip_for_outbound_sockets=127.8.29.193
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:45925
--builtin_ntp_servers=127.8.29.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 17:33:14.877676 28242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:14.877983 28242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:14.878103 28242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:14.885581 28242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:14.885694 28242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:14.885823 28242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.193
I20260923 17:33:14.895215 28242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.29.193
--webserver_port=0
--tserver_master_addrs=127.8.29.254:45925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.193
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:14.896522 28242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:14.898245 28242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:14.911755 28247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:14.911835 28248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:14.911789 28250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:14.912616 28242 server_base.cc:1034] running on GCE node
I20260923 17:33:14.913221 28242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:14.914165 28242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:14.915375 28242 hybrid_clock.cc:648] HybridClock initialized: now 1790184794915314 us; error 48 us; skew 500 ppm
I20260923 17:33:14.915700 28242 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:14.918006 28242 webserver.cc:460] Webserver started at http://127.8.29.193:40525/ using document root <none> and password file <none>
I20260923 17:33:14.918753 28242 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:14.918846 28242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:14.919147 28242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:14.922112 28242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/data/instance:
uuid: "52884c8833eb461095e92afe19b07f20"
format_stamp: "Formatted at 2026-09-23 17:33:14 on dist-test-slave-p7jt"
server_key: "c4e51f6433295cba9ddce2f807e150db"
server_key_iv: "72948ca9902094553d9bab7ae63246b1"
server_key_version: "encryptionkey@0"
I20260923 17:33:14.922981 28242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance:
uuid: "52884c8833eb461095e92afe19b07f20"
format_stamp: "Formatted at 2026-09-23 17:33:14 on dist-test-slave-p7jt"
server_key: "c4e51f6433295cba9ddce2f807e150db"
server_key_iv: "72948ca9902094553d9bab7ae63246b1"
server_key_version: "encryptionkey@0"
I20260923 17:33:14.928490 28242 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:33:14.932286 28256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:14.933925 28242 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:33:14.934128 28242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/wal
uuid: "52884c8833eb461095e92afe19b07f20"
format_stamp: "Formatted at 2026-09-23 17:33:14 on dist-test-slave-p7jt"
server_key: "c4e51f6433295cba9ddce2f807e150db"
server_key_iv: "72948ca9902094553d9bab7ae63246b1"
server_key_version: "encryptionkey@0"
I20260923 17:33:14.934423 28242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:14.957247 28242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:14.958071 28242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:14.958336 28242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:14.959471 28242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:14.961015 28242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:14.961102 28242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:14.961174 28242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:14.961225 28242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:14.992166 28242 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.193:45141
I20260923 17:33:14.992285 28368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.193:45141 every 8 connection(s)
I20260923 17:33:14.993494 28242 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/data/info.pb
I20260923 17:33:15.000515 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 28242
I20260923 17:33:15.000705 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/wal/instance
I20260923 17:33:15.001139 8311 external_mini_cluster.cc:1428] Setting key eecf354e19037690b7f6c8d22dcb7af1
I20260923 17:33:15.004105 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.194:0
--local_ip_for_outbound_sockets=127.8.29.194
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:45925
--builtin_ntp_servers=127.8.29.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:33:15.007669 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:14.996486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.193:39865 (local address 127.8.29.254:45925)
0923 17:33:14.996980 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:14.996985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:14.998284 (+ 1299us) server_negotiation.cc:407] Connection header received
0923 17:33:15.000399 (+ 2115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:15.000406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:15.000505 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:15.000641 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:15.001990 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.002848 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.004708 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.005341 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.005467 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.006871 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:15.006876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:15.006881 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:15.006903 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:15.006949 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:15.006964 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:15.007048 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:15.007234 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 17:33:15.007335 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":265,"threads_started":1}
I20260923 17:33:15.008679 28372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:14.997092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:45925 (local address 127.8.29.193:39865)
0923 17:33:14.998106 (+ 1014us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:14.998144 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:15.000029 (+ 1885us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:15.000968 (+ 939us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:15.000978 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:15.001092 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:15.001785 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.001806 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.003030 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.003036 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.004542 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.004558 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.005552 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.005557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.005783 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.006515 (+ 732us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:33:15.006518 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:15.006566 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:33:15.006663 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:15.007051 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:15.007058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:15.007064 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 17:33:15.007208 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 17:33:15.007567 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":301,"threads_started":1}
I20260923 17:33:15.010263 28369 heartbeater.cc:347] Connected to a master server at 127.8.29.254:45925
I20260923 17:33:15.010620 28369 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:15.011402 28369 heartbeater.cc:511] Master 127.8.29.254:45925 requested a full tablet report, sending...
I20260923 17:33:15.013805 28183 ts_manager.cc:194] Registered new tserver with Master: 52884c8833eb461095e92afe19b07f20 (127.8.29.193:45141)
I20260923 17:33:15.015403 28183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.193:39865
W20260923 17:33:15.330551 28373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:15.330843 28373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:15.330968 28373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:15.338438 28373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:15.338567 28373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:15.338747 28373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.194
I20260923 17:33:15.348040 28373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.29.194
--webserver_port=0
--tserver_master_addrs=127.8.29.254:45925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.194
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:15.349326 28373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:15.351099 28373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:15.364346 28378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:15.364385 28379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:15.364351 28381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:15.370697 28373 server_base.cc:1034] running on GCE node
I20260923 17:33:15.371366 28373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:15.372366 28373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:15.373590 28373 hybrid_clock.cc:648] HybridClock initialized: now 1790184795373525 us; error 53 us; skew 500 ppm
I20260923 17:33:15.373922 28373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:15.376236 28373 webserver.cc:460] Webserver started at http://127.8.29.194:32879/ using document root <none> and password file <none>
I20260923 17:33:15.376919 28373 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:15.377013 28373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:15.377327 28373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:15.380437 28373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/data/instance:
uuid: "b75e08563ed949e18d4e537c37121445"
format_stamp: "Formatted at 2026-09-23 17:33:15 on dist-test-slave-p7jt"
server_key: "5608ac97aa5e8bc85dfed9c72f210725"
server_key_iv: "df699d52b46809a78828f0803ff4b681"
server_key_version: "encryptionkey@0"
I20260923 17:33:15.381299 28373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance:
uuid: "b75e08563ed949e18d4e537c37121445"
format_stamp: "Formatted at 2026-09-23 17:33:15 on dist-test-slave-p7jt"
server_key: "5608ac97aa5e8bc85dfed9c72f210725"
server_key_iv: "df699d52b46809a78828f0803ff4b681"
server_key_version: "encryptionkey@0"
I20260923 17:33:15.386600 28373 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 17:33:15.390025 28387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:15.391564 28373 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:15.391726 28373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/wal
uuid: "b75e08563ed949e18d4e537c37121445"
format_stamp: "Formatted at 2026-09-23 17:33:15 on dist-test-slave-p7jt"
server_key: "5608ac97aa5e8bc85dfed9c72f210725"
server_key_iv: "df699d52b46809a78828f0803ff4b681"
server_key_version: "encryptionkey@0"
I20260923 17:33:15.391955 28373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:15.418195 28373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:15.419107 28373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:15.419404 28373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:15.420507 28373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:15.422013 28373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:15.422108 28373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:15.422223 28373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:15.422297 28373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:15.453655 28373 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.194:33763
I20260923 17:33:15.453706 28499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.194:33763 every 8 connection(s)
I20260923 17:33:15.455077 28373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/data/info.pb
I20260923 17:33:15.460999 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 28373
I20260923 17:33:15.461201 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/wal/instance
I20260923 17:33:15.461722 8311 external_mini_cluster.cc:1428] Setting key 7c2286bd8074a1e277d4f3ed050b2d0f
I20260923 17:33:15.464768 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.29.195:0
--local_ip_for_outbound_sockets=127.8.29.195
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:45925
--builtin_ntp_servers=127.8.29.212:39657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 17:33:15.468393 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.457708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:32941 (local address 127.8.29.254:45925)
0923 17:33:15.457893 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:15.457899 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:15.459428 (+ 1529us) server_negotiation.cc:407] Connection header received
0923 17:33:15.461498 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:15.461503 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:15.461577 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:15.461663 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:15.463032 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.463914 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.465708 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.466121 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.466232 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.467619 (+ 1387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:15.467624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:15.467628 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:15.467646 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:15.467678 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:15.467690 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:15.467762 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:15.467991 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 17:33:15.468096 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260923 17:33:15.469470 28502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.458280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:45925 (local address 127.8.29.194:32941)
0923 17:33:15.459278 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:15.459315 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:15.461169 (+ 1854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:15.461952 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:15.461964 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:15.462101 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:15.462845 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.462871 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.464050 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.464054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.465563 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.465576 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.466358 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.466367 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.466630 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.467279 (+ 649us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:33:15.467282 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:15.467334 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:33:15.467423 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:15.467773 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:15.467785 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:15.467794 (+ 9us) client_negotiation.cc:769] Sending connection context
0923 17:33:15.467949 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 17:33:15.468398 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":315,"threads_started":1}
I20260923 17:33:15.470916 28500 heartbeater.cc:347] Connected to a master server at 127.8.29.254:45925
I20260923 17:33:15.471211 28500 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:15.471951 28500 heartbeater.cc:511] Master 127.8.29.254:45925 requested a full tablet report, sending...
I20260923 17:33:15.473362 28183 ts_manager.cc:194] Registered new tserver with Master: b75e08563ed949e18d4e537c37121445 (127.8.29.194:33763)
I20260923 17:33:15.474161 28183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.194:32941
W20260923 17:33:15.789479 28503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:15.789796 28503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:15.789927 28503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:15.797330 28503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:15.797446 28503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:15.797595 28503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.29.195
I20260923 17:33:15.806761 28503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.29.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.29.195
--webserver_port=0
--tserver_master_addrs=127.8.29.254:45925
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.29.195
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:15.808079 28503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:15.809886 28503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:15.822710 28508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:15.822634 28509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:15.822665 28511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:15.823772 28503 server_base.cc:1034] running on GCE node
I20260923 17:33:15.824462 28503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:15.825501 28503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:15.826700 28503 hybrid_clock.cc:648] HybridClock initialized: now 1790184795826640 us; error 56 us; skew 500 ppm
I20260923 17:33:15.827016 28503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260923 17:33:15.829242 28503 webserver.cc:460] Webserver started at http://127.8.29.195:36373/ using document root <none> and password file <none>
I20260923 17:33:15.829941 28503 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:15.830044 28503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:15.830358 28503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:15.833289 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/data/instance:
uuid: "7625c5d402014a23887cb6a46fa4aa2c"
format_stamp: "Formatted at 2026-09-23 17:33:15 on dist-test-slave-p7jt"
server_key: "3183422e2e7088475d71d72a51ef38f2"
server_key_iv: "4a24e93401a5eaebe187f6cd63322bee"
server_key_version: "encryptionkey@0"
I20260923 17:33:15.834049 28503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance:
uuid: "7625c5d402014a23887cb6a46fa4aa2c"
format_stamp: "Formatted at 2026-09-23 17:33:15 on dist-test-slave-p7jt"
server_key: "3183422e2e7088475d71d72a51ef38f2"
server_key_iv: "4a24e93401a5eaebe187f6cd63322bee"
server_key_version: "encryptionkey@0"
I20260923 17:33:15.839458 28503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:33:15.843061 28517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:15.844580 28503 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:15.844743 28503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/wal
uuid: "7625c5d402014a23887cb6a46fa4aa2c"
format_stamp: "Formatted at 2026-09-23 17:33:15 on dist-test-slave-p7jt"
server_key: "3183422e2e7088475d71d72a51ef38f2"
server_key_iv: "4a24e93401a5eaebe187f6cd63322bee"
server_key_version: "encryptionkey@0"
I20260923 17:33:15.844974 28503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:15.867834 28503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:15.868702 28503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:15.869009 28503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:15.869982 28503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 17:33:15.871517 28503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 17:33:15.871613 28503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:15.871716 28503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 17:33:15.871790 28503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:15.900745 28503 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.195:43617
I20260923 17:33:15.900811 28629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.195:43617 every 8 connection(s)
I20260923 17:33:15.902029 28503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/data/info.pb
I20260923 17:33:15.908929 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 28503
I20260923 17:33:15.909078 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/wal/instance
I20260923 17:33:15.909502 8311 external_mini_cluster.cc:1428] Setting key 1ba96804045aa26d775bfd007bc512d8
I20260923 17:33:15.914876 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.904666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.195:58139 (local address 127.8.29.254:45925)
0923 17:33:15.904895 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:15.904900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:15.906343 (+ 1443us) server_negotiation.cc:407] Connection header received
0923 17:33:15.908383 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:15.908389 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:15.908472 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:15.908558 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:15.909763 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.910723 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.912266 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.912691 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.912799 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.914064 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:15.914070 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:15.914073 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:15.914092 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:15.914122 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:15.914134 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:15.914204 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:15.914412 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 17:33:15.914536 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260923 17:33:15.915853 28632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.905169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:45925 (local address 127.8.29.195:58139)
0923 17:33:15.906161 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:15.906199 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:15.908068 (+ 1869us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:15.908826 (+ 758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:15.908835 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:15.908944 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:15.909596 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.909617 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.910872 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.910876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.912126 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.912139 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.912908 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.912915 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.913137 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.913719 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 17:33:15.913722 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:15.913762 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 17:33:15.913886 (+ 124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:15.914216 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:15.914227 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:15.914234 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 17:33:15.914390 (+ 156us) client_negotiation.cc:240] Negotiation successful
0923 17:33:15.914839 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":296,"threads_started":1}
I20260923 17:33:15.917096 28630 heartbeater.cc:347] Connected to a master server at 127.8.29.254:45925
I20260923 17:33:15.917410 28630 heartbeater.cc:464] Registering TS with master...
I20260923 17:33:15.918174 28630 heartbeater.cc:511] Master 127.8.29.254:45925 requested a full tablet report, sending...
I20260923 17:33:15.919620 28183 ts_manager.cc:194] Registered new tserver with Master: 7625c5d402014a23887cb6a46fa4aa2c (127.8.29.195:43617)
I20260923 17:33:15.920406 28183 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.29.195:58139
I20260923 17:33:15.924335 8311 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 17:33:15.933887 28371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.928322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51304 (local address 127.8.29.254:45925)
0923 17:33:15.928548 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:15.928554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:15.928861 (+ 307us) server_negotiation.cc:407] Connection header received
0923 17:33:15.929008 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:15.929012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:15.929084 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:15.929164 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:15.929628 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.930320 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.931660 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.932080 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.932188 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.933176 (+ 988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:15.933180 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:15.933184 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:15.933241 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:15.933276 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:15.933290 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:15.933371 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:15.933496 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 17:33:15.933593 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:33:15.937603 28183 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 17:33:15.944523 28183 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 17:33:15.969415 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.957196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54538 (local address 127.8.29.193:45141)
0923 17:33:15.957751 (+ 555us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:15.957758 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:15.958562 (+ 804us) server_negotiation.cc:407] Connection header received
0923 17:33:15.959117 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:15.959125 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:15.959431 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:15.959616 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:15.960715 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.962502 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.965823 (+ 3321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.967507 (+ 1684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.967631 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.967922 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:15.968767 (+ 845us) server_negotiation.cc:299] Negotiation successful
0923 17:33:15.968958 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":404,"thread_start_us":284,"threads_started":1}
I20260923 17:33:15.970129 28642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.957073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:45141 (local address 127.0.0.1:54538)
0923 17:33:15.958305 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:15.958363 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:15.958909 (+ 546us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:15.960004 (+ 1095us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:15.960013 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:15.960057 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:15.960496 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.960509 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.962751 (+ 2242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.962757 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.965532 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.965555 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.968146 (+ 2591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.968155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.968464 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.968526 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 17:33:15.969549 (+ 1023us) client_negotiation.cc:240] Negotiation successful
0923 17:33:15.969806 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":946,"thread_start_us":805,"threads_started":1}
I20260923 17:33:15.974198 28304 tablet_service.cc:1467] Processing CreateTablet for tablet fdf2bcf699d046f9b966e33c8432ee49 (DEFAULT_TABLE table=test-table [id=7972082db9a84e1f8e6b6a7afe054e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:15.975813 28304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf2bcf699d046f9b966e33c8432ee49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:15.977039 28647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.959079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.194:33763 (local address 127.0.0.1:59270)
0923 17:33:15.967647 (+ 8568us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:15.967666 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:15.967793 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:15.969551 (+ 1758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:15.969556 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:15.969585 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:15.969958 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.969970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.971985 (+ 2015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.971991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.974381 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.974397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.976222 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.976229 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.976408 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.976444 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 17:33:15.976516 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 17:33:15.976658 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8439,"mutex_wait_us":113,"thread_start_us":8806,"threads_started":1}
I20260923 17:33:15.977420 28644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.957791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:43617 (local address 127.0.0.1:52870)
0923 17:33:15.958341 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:15.958362 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:15.958582 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:15.968966 (+ 10384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:15.968971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:15.969003 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:15.969379 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.969391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.972542 (+ 3151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.972547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.974657 (+ 2110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:15.974669 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.976644 (+ 1975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.976650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:15.976837 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.976863 (+ 26us) client_negotiation.cc:769] Sending connection context
0923 17:33:15.976961 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 17:33:15.977069 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"mutex_wait_us":23,"thread_start_us":727,"threads_started":1}
I20260923 17:33:15.978049 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.958338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52870 (local address 127.8.29.195:43617)
0923 17:33:15.967321 (+ 8983us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:15.967331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:15.967371 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:33:15.967485 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:15.967493 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:15.967802 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:15.968037 (+ 235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:15.969806 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.972355 (+ 2549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.974835 (+ 2480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.976107 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.976237 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.976515 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:15.977644 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0923 17:33:15.977791 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8800,"thread_start_us":8650,"threads_started":1}
I20260923 17:33:15.979398 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:15.959205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.8.29.194:33763)
0923 17:33:15.968864 (+ 9659us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:15.968872 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:15.968906 (+ 34us) server_negotiation.cc:407] Connection header received
0923 17:33:15.969003 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:15.969011 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:15.969318 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:15.969818 (+ 500us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:15.970639 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.971793 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.974597 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:15.975991 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:15.977421 (+ 1430us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:15.977816 (+ 395us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:15.977901 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 17:33:15.978066 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9479,"thread_start_us":9360,"threads_started":1}
I20260923 17:33:15.982056 28565 tablet_service.cc:1467] Processing CreateTablet for tablet fdf2bcf699d046f9b966e33c8432ee49 (DEFAULT_TABLE table=test-table [id=7972082db9a84e1f8e6b6a7afe054e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:15.983570 28565 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf2bcf699d046f9b966e33c8432ee49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:15.983187 28435 tablet_service.cc:1467] Processing CreateTablet for tablet fdf2bcf699d046f9b966e33c8432ee49 (DEFAULT_TABLE table=test-table [id=7972082db9a84e1f8e6b6a7afe054e70]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 17:33:15.984560 28435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fdf2bcf699d046f9b966e33c8432ee49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:15.991237 28648 tablet_bootstrap.cc:492] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: Bootstrap starting.
I20260923 17:33:15.997558 28648 tablet_bootstrap.cc:654] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:15.999138 28648 log.cc:824] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:16.002619 28649 tablet_bootstrap.cc:492] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: Bootstrap starting.
I20260923 17:33:16.008544 28650 tablet_bootstrap.cc:492] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: Bootstrap starting.
I20260923 17:33:16.006407 28649 tablet_bootstrap.cc:654] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:16.010592 28649 log.cc:824] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:16.011665 28648 tablet_bootstrap.cc:492] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: No bootstrap required, opened a new log
I20260923 17:33:16.011809 28650 tablet_bootstrap.cc:654] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:16.011977 28648 ts_tablet_manager.cc:1397] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.008s
I20260923 17:33:16.013049 28650 log.cc:824] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:16.013968 28649 tablet_bootstrap.cc:492] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: No bootstrap required, opened a new log
I20260923 17:33:16.014205 28649 ts_tablet_manager.cc:1397] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260923 17:33:16.016122 28650 tablet_bootstrap.cc:492] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: No bootstrap required, opened a new log
I20260923 17:33:16.016392 28650 ts_tablet_manager.cc:1397] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20260923 17:33:16.018617 28369 heartbeater.cc:503] Master 127.8.29.254:45925 was elected leader, sending a full tablet report...
I20260923 17:33:16.020649 28648 raft_consensus.cc:348] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } }
I20260923 17:33:16.021111 28648 raft_consensus.cc:374] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:16.021203 28648 raft_consensus.cc:729] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52884c8833eb461095e92afe19b07f20, State: Initialized, Role: FOLLOWER
I20260923 17:33:16.021816 28648 consensus_queue.cc:260] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } }
I20260923 17:33:16.024703 28649 raft_consensus.cc:348] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } }
I20260923 17:33:16.025156 28649 raft_consensus.cc:374] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:16.025308 28649 raft_consensus.cc:729] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7625c5d402014a23887cb6a46fa4aa2c, State: Initialized, Role: FOLLOWER
I20260923 17:33:16.026029 28649 consensus_queue.cc:260] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } }
I20260923 17:33:16.026520 28650 raft_consensus.cc:348] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } }
I20260923 17:33:16.026993 28650 raft_consensus.cc:374] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:16.026985 28648 ts_tablet_manager.cc:1428] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260923 17:33:16.027082 28650 raft_consensus.cc:729] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b75e08563ed949e18d4e537c37121445, State: Initialized, Role: FOLLOWER
I20260923 17:33:16.027699 28650 consensus_queue.cc:260] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } }
I20260923 17:33:16.028862 28500 heartbeater.cc:503] Master 127.8.29.254:45925 was elected leader, sending a full tablet report...
I20260923 17:33:16.029529 28650 ts_tablet_manager.cc:1428] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: Time spent starting tablet: real 0.013s user 0.011s sys 0.003s
I20260923 17:33:16.030213 28630 heartbeater.cc:503] Master 127.8.29.254:45925 was elected leader, sending a full tablet report...
I20260923 17:33:16.030982 28649 ts_tablet_manager.cc:1428] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
W20260923 17:33:16.153635 28631 tablet.cc:2369] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:33:16.207221 28501 tablet.cc:2369] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:16.212512 28656 raft_consensus.cc:482] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 17:33:16.212816 28656 raft_consensus.cc:504] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } }
I20260923 17:33:16.214424 28656 leader_election.cc:288] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52884c8833eb461095e92afe19b07f20 (127.8.29.193:45141), 7625c5d402014a23887cb6a46fa4aa2c (127.8.29.195:43617)
I20260923 17:33:16.222903 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.214877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:33835 (local address 127.8.29.195:43617)
0923 17:33:16.215044 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:16.215049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:16.215779 (+ 730us) server_negotiation.cc:407] Connection header received
0923 17:33:16.215916 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:16.215922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:16.215994 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:16.216073 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:16.216763 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.217498 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.220518 (+ 3020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.221766 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.221948 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:16.222006 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:16.222491 (+ 485us) server_negotiation.cc:299] Negotiation successful
0923 17:33:16.222637 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:33:16.222903 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.214810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.29.194:54013 (local address 127.8.29.193:45141)
0923 17:33:16.215000 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:16.215006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:16.215711 (+ 705us) server_negotiation.cc:407] Connection header received
0923 17:33:16.215934 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:16.215938 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:16.216011 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:16.216064 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 17:33:16.216858 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.217530 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.220516 (+ 2986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.221747 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.221918 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:16.221993 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:16.222489 (+ 496us) server_negotiation.cc:299] Negotiation successful
0923 17:33:16.222637 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:33:16.222972 28657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.214680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:45141 (local address 127.8.29.194:54013)
0923 17:33:16.215577 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:16.215590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:16.215737 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:16.216258 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:16.216262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:16.216296 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:16.216694 (+ 398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:16.216704 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.217673 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.217678 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:16.220254 (+ 2576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:16.220269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.221991 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.221997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:16.222230 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:16.222281 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 17:33:16.222376 (+ 95us) client_negotiation.cc:240] Negotiation successful
0923 17:33:16.222559 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":342,"thread_start_us":413,"threads_started":1}
I20260923 17:33:16.223026 28658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.214680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.195:43617 (local address 127.8.29.194:33835)
0923 17:33:16.215555 (+ 875us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:16.215582 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:16.215778 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:16.216125 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:16.216130 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:16.216156 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 17:33:16.216590 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:16.216606 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.217644 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.217649 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:16.220251 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:16.220269 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.221986 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.221994 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:16.222230 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:16.222279 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 17:33:16.222376 (+ 97us) client_negotiation.cc:240] Negotiation successful
0923 17:33:16.222559 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"mutex_wait_us":170,"spinlock_wait_cycles":38784,"thread_start_us":512,"threads_started":1}
I20260923 17:33:16.224148 28585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf2bcf699d046f9b966e33c8432ee49" candidate_uuid: "b75e08563ed949e18d4e537c37121445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7625c5d402014a23887cb6a46fa4aa2c" is_pre_election: true
I20260923 17:33:16.224148 28324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf2bcf699d046f9b966e33c8432ee49" candidate_uuid: "b75e08563ed949e18d4e537c37121445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52884c8833eb461095e92afe19b07f20" is_pre_election: true
I20260923 17:33:16.224761 28585 raft_consensus.cc:2454] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75e08563ed949e18d4e537c37121445 in term 0.
I20260923 17:33:16.224761 28324 raft_consensus.cc:2454] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b75e08563ed949e18d4e537c37121445 in term 0.
I20260923 17:33:16.225634 28390 leader_election.cc:302] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 52884c8833eb461095e92afe19b07f20, b75e08563ed949e18d4e537c37121445; no voters:
I20260923 17:33:16.226162 28656 raft_consensus.cc:2781] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 17:33:16.226303 28656 raft_consensus.cc:482] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 17:33:16.226375 28656 raft_consensus.cc:3037] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:16.228749 28656 raft_consensus.cc:504] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } }
I20260923 17:33:16.229368 28656 leader_election.cc:288] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [CANDIDATE]: Term 1 election: Requested vote from peers 52884c8833eb461095e92afe19b07f20 (127.8.29.193:45141), 7625c5d402014a23887cb6a46fa4aa2c (127.8.29.195:43617)
I20260923 17:33:16.229950 28324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf2bcf699d046f9b966e33c8432ee49" candidate_uuid: "b75e08563ed949e18d4e537c37121445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52884c8833eb461095e92afe19b07f20"
I20260923 17:33:16.229972 28585 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fdf2bcf699d046f9b966e33c8432ee49" candidate_uuid: "b75e08563ed949e18d4e537c37121445" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7625c5d402014a23887cb6a46fa4aa2c"
I20260923 17:33:16.230158 28324 raft_consensus.cc:3037] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:16.230171 28585 raft_consensus.cc:3037] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:16.232470 28324 raft_consensus.cc:2454] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b75e08563ed949e18d4e537c37121445 in term 1.
I20260923 17:33:16.232473 28585 raft_consensus.cc:2454] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b75e08563ed949e18d4e537c37121445 in term 1.
I20260923 17:33:16.232969 28390 leader_election.cc:302] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 52884c8833eb461095e92afe19b07f20, 7625c5d402014a23887cb6a46fa4aa2c, b75e08563ed949e18d4e537c37121445; no voters:
I20260923 17:33:16.233317 28656 raft_consensus.cc:2781] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:16.233752 28656 raft_consensus.cc:686] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 1 LEADER]: Becoming Leader. State: Replica: b75e08563ed949e18d4e537c37121445, State: Running, Role: LEADER
I20260923 17:33:16.234220 28656 consensus_queue.cc:237] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } }
I20260923 17:33:16.239447 28183 catalog_manager.cc:5495] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 reported cstate change: term changed from 0 to 1, leader changed from <none> to b75e08563ed949e18d4e537c37121445 (127.8.29.194). New cstate: current_term: 1 leader_uuid: "b75e08563ed949e18d4e537c37121445" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b75e08563ed949e18d4e537c37121445" member_type: VOTER last_known_addr { host: "127.8.29.194" port: 33763 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 } health_report { overall_health: UNKNOWN } } }
W20260923 17:33:16.245920 28370 tablet.cc:2369] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:16.280536 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.275237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59274 (local address 127.8.29.194:33763)
0923 17:33:16.275470 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:16.275475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:16.275528 (+ 53us) server_negotiation.cc:407] Connection header received
0923 17:33:16.275618 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:16.275622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:16.275716 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:16.275841 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:16.276219 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.276931 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.278296 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.278774 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.278878 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:16.279352 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:16.279967 (+ 615us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:16.280075 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:16.280211 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 17:33:16.280284 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260923 17:33:16.289132 28324 raft_consensus.cc:1261] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [term 1 FOLLOWER]: Refusing update from remote peer b75e08563ed949e18d4e537c37121445: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:16.289132 28585 raft_consensus.cc:1261] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [term 1 FOLLOWER]: Refusing update from remote peer b75e08563ed949e18d4e537c37121445: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:16.290548 28659 consensus_queue.cc:1035] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7625c5d402014a23887cb6a46fa4aa2c" member_type: VOTER last_known_addr { host: "127.8.29.195" port: 43617 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:16.290951 28656 consensus_queue.cc:1035] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52884c8833eb461095e92afe19b07f20" member_type: VOTER last_known_addr { host: "127.8.29.193" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:16.302829 28662 mvcc.cc:205] Tried to move back new op lower bound from 7332596925588692992 to 7332596925379596288. Current Snapshot: MvccSnapshot[applied={T|T < 7332596925588692992}]
I20260923 17:33:16.304243 28664 mvcc.cc:205] Tried to move back new op lower bound from 7332596925588692992 to 7332596925379596288. Current Snapshot: MvccSnapshot[applied={T|T < 7332596925588692992}]
I20260923 17:33:16.320935 28183 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:16.321195 28183 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 17:33:16.324717 28183 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f6e4870f0cfb476db5f797c841139943: Sending DeleteTablet for 3 replicas of tablet fdf2bcf699d046f9b966e33c8432ee49
I20260923 17:33:16.325946 28565 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf2bcf699d046f9b966e33c8432ee49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:16 UTC) from {username='slave'} at 127.0.0.1:52870
I20260923 17:33:16.326301 28435 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf2bcf699d046f9b966e33c8432ee49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:16 UTC) from {username='slave'} at 127.0.0.1:59270
I20260923 17:33:16.326892 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 28242
I20260923 17:33:16.327064 28304 tablet_service.cc:1514] Processing DeleteTablet for tablet fdf2bcf699d046f9b966e33c8432ee49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 17:33:16 UTC) from {username='slave'} at 127.0.0.1:54538
I20260923 17:33:16.338632 28673 tablet_replica.cc:330] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: stopping tablet replica
I20260923 17:33:16.339184 28673 raft_consensus.cc:2229] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 1 LEADER]: Raft consensus shutting down.
I20260923 17:33:16.339932 28673 raft_consensus.cc:2258] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:16.340758 28674 tablet_replica.cc:330] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: stopping tablet replica
I20260923 17:33:16.341111 28672 tablet_replica.cc:330] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: stopping tablet replica
I20260923 17:33:16.341619 28672 raft_consensus.cc:2229] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:16.342204 28672 raft_consensus.cc:2258] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:16.343201 28673 ts_tablet_manager.cc:1905] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:16.344769 28672 ts_tablet_manager.cc:1905] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:16.345353 28674 raft_consensus.cc:2229] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 17:33:16.345906 28674 raft_consensus.cc:2258] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 17:33:16.350068 28674 ts_tablet_manager.cc:1905] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 17:33:16.350767 28673 ts_tablet_manager.cc:1918] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:16.350942 28673 log.cc:1196] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-1/wal/wals/fdf2bcf699d046f9b966e33c8432ee49
I20260923 17:33:16.351457 28673 ts_tablet_manager.cc:1939] T fdf2bcf699d046f9b966e33c8432ee49 P b75e08563ed949e18d4e537c37121445: Deleting consensus metadata
I20260923 17:33:16.351987 28672 ts_tablet_manager.cc:1918] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:16.352216 28672 log.cc:1196] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-2/wal/wals/fdf2bcf699d046f9b966e33c8432ee49
I20260923 17:33:16.352768 28672 ts_tablet_manager.cc:1939] T fdf2bcf699d046f9b966e33c8432ee49 P 7625c5d402014a23887cb6a46fa4aa2c: Deleting consensus metadata
I20260923 17:33:16.353749 28169 catalog_manager.cc:4841] TS b75e08563ed949e18d4e537c37121445 (127.8.29.194:33763): tablet fdf2bcf699d046f9b966e33c8432ee49 (table test-table [id=7972082db9a84e1f8e6b6a7afe054e70]) successfully deleted
I20260923 17:33:16.354780 28167 catalog_manager.cc:4841] TS 7625c5d402014a23887cb6a46fa4aa2c (127.8.29.195:43617): tablet fdf2bcf699d046f9b966e33c8432ee49 (table test-table [id=7972082db9a84e1f8e6b6a7afe054e70]) successfully deleted
I20260923 17:33:16.356195 28643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.328233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.8.29.193:45141)
0923 17:33:16.328415 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:16.328421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:16.337768 (+ 9347us) server_negotiation.cc:407] Connection header received
0923 17:33:16.341141 (+ 3373us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:16.341148 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:16.341260 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:16.341374 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:16.342096 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.343262 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.346323 (+ 3061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.347002 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.347106 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:16.354577 (+ 7471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:16.354589 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:16.354608 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:16.355391 (+ 783us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:16.355448 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:16.355471 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:16.355589 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:16.355760 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 17:33:16.355880 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:33:16.356907 28364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:16.456732 28674 ts_tablet_manager.cc:1918] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 17:33:16.456974 28674 log.cc:1196] T fdf2bcf699d046f9b966e33c8432ee49 P 52884c8833eb461095e92afe19b07f20: Deleting WAL directory at /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790184478467572-8311-0/minicluster-data/ts-0/wal/wals/fdf2bcf699d046f9b966e33c8432ee49
I20260923 17:33:16.457059 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 28242
W20260923 17:33:16.478901 28169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.29.193:45141 (error 108)
I20260923 17:33:16.479558 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 28373
I20260923 17:33:16.483853 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.482277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:45141 (local address 127.0.0.1:54548)
0923 17:33:16.483527 (+ 1250us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:16.483657 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1119,"threads_started":1}
W20260923 17:33:16.485176 28169 catalog_manager.cc:4584] TS 52884c8833eb461095e92afe19b07f20 (127.8.29.193:45141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fdf2bcf699d046f9b966e33c8432ee49: Network error: Client connection negotiation failed: client connection to 127.8.29.193:45141: connect: Connection refused (error 111)
I20260923 17:33:16.488313 28646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.480321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59288 (local address 127.8.29.194:33763)
0923 17:33:16.480944 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:16.480950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:16.480968 (+ 18us) server_negotiation.cc:407] Connection header received
0923 17:33:16.481031 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:16.481036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:16.481133 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:16.481231 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:16.481879 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.483125 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.484876 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.485525 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.485624 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:16.486794 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:16.486804 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:16.486821 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:16.487497 (+ 676us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:16.487552 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:16.487572 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:16.487658 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:16.487870 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 17:33:16.487965 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485}
I20260923 17:33:16.489010 28495 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:16.538846 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.538266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:45141 (local address 127.0.0.1:54558)
0923 17:33:16.538546 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:16.538666 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260923 17:33:16.539903 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.539506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:45141 (local address 127.0.0.1:54572)
0923 17:33:16.539659 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:16.539743 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 17:33:16.600679 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.600111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:45141 (local address 127.0.0.1:54586)
0923 17:33:16.600364 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:16.600480 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260923 17:33:16.601950 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.601458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:45141 (local address 127.0.0.1:54602)
0923 17:33:16.601632 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:16.601727 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260923 17:33:16.608445 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 28373
I20260923 17:33:16.624408 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 28503
I20260923 17:33:16.630666 28645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.625098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.8.29.195:43617)
0923 17:33:16.625257 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:16.625263 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:16.625334 (+ 71us) server_negotiation.cc:407] Connection header received
0923 17:33:16.625475 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:16.625479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:16.625556 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:16.625656 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:16.626018 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.626777 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.628019 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:16.628380 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:16.628467 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:16.629414 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:16.629422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:16.629437 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:16.629999 (+ 562us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:16.630047 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:16.630067 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:16.630165 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:16.630268 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 17:33:16.630354 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:33:16.631245 28625 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:16.682665 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.682085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:45141 (local address 127.0.0.1:50164)
0923 17:33:16.682326 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:16.682481 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 17:33:16.683858 28678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:16.683364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.193:45141 (local address 127.0.0.1:50172)
0923 17:33:16.683596 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:16.683685 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.193:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260923 17:33:16.719806 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 28503
I20260923 17:33:16.732939 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 28152
I20260923 17:33:16.734026 28213 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:16.847827 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 28152
2026-09-23T17:33:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2926 ms)
[----------] 6 tests from EncryptionPolicyTest (21467 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:16 dist-test-slave-p7jt krb5kdc[28684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:16 dist-test-slave-p7jt krb5kdc[28684](info): set up 2 sockets
Sep 23 17:33:16 dist-test-slave-p7jt krb5kdc[28684](info): commencing operation
krb5kdc: starting...
W20260923 17:33:17.874863 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.008s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:18 dist-test-slave-p7jt krb5kdc[28684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:18.128525 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37791
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37791
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:18.459195 28697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:18.459519 28697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:18.459663 28697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:18.467226 28697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:18.467350 28697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:18.467407 28697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:18.467490 28697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:18.467545 28697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:18.477710 28697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37791
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:18.479355 28697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:18.480770 28697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:18.492374 28702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:18.492630 28703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:18.492684 28705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:18.493490 28697 server_base.cc:1034] running on GCE node
I20260923 17:33:18.494293 28697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:18.495841 28697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:18.497051 28697 hybrid_clock.cc:648] HybridClock initialized: now 1790184798497000 us; error 43 us; skew 500 ppm
I20260923 17:33:18.497414 28697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:18 dist-test-slave-p7jt krb5kdc[28684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:18.508605 28697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:18.510320 28697 webserver.cc:460] Webserver started at http://127.0.0.1:38667/ using document root <none> and password file <none>
I20260923 17:33:18.511106 28697 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:18.511229 28697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:18.511520 28697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:18.514613 28697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "638b83033c544bbb91810e1e0116c814"
format_stamp: "Formatted at 2026-09-23 17:33:18 on dist-test-slave-p7jt"
server_key: "f8209340695c138ccf771a27298ef8dd"
server_key_iv: "282cca5de160d855e8fa4b3a3096ade3"
server_key_version: "encryptionkey@0"
I20260923 17:33:18.515394 28697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "638b83033c544bbb91810e1e0116c814"
format_stamp: "Formatted at 2026-09-23 17:33:18 on dist-test-slave-p7jt"
server_key: "f8209340695c138ccf771a27298ef8dd"
server_key_iv: "282cca5de160d855e8fa4b3a3096ade3"
server_key_version: "encryptionkey@0"
I20260923 17:33:18.521332 28697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260923 17:33:18.526039 28712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:18.527607 28697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:18.527799 28697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "638b83033c544bbb91810e1e0116c814"
format_stamp: "Formatted at 2026-09-23 17:33:18 on dist-test-slave-p7jt"
server_key: "f8209340695c138ccf771a27298ef8dd"
server_key_iv: "282cca5de160d855e8fa4b3a3096ade3"
server_key_version: "encryptionkey@0"
I20260923 17:33:18.528033 28697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:18.564499 28697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:18.569106 28697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:18.569438 28697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:18.588562 28697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37791
I20260923 17:33:18.588585 28763 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37791 every 8 connection(s)
I20260923 17:33:18.590096 28697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:18.593132 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 28697
I20260923 17:33:18.593343 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:18.594094 8311 external_mini_cluster.cc:1428] Setting key d20ab96a437639a6e55d300d03a4d2f7
I20260923 17:33:18.594502 28764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:18.608925 28764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814: Bootstrap starting.
Sep 23 17:33:18 dist-test-slave-p7jt krb5kdc[28684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184798, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:33:18.612138 28764 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:18.613212 28764 log.cc:824] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:18.615967 28764 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814: No bootstrap required, opened a new log
I20260923 17:33:18.617591 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:18.596091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47561 (local address 127.0.0.1:37791)
0923 17:33:18.596879 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:18.596891 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:18.596933 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:33:18.599115 (+ 2182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:18.599139 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:18.599616 (+ 477us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:18.599981 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:18.601819 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:18.603165 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:18.604855 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:18.605560 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:18.605702 (+ 142us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:18.610444 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:18.610480 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:18.610501 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:18.610549 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:18.613474 (+ 2925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:18.614312 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:18.614319 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:18.614325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:18.614536 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:18.614990 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:18.614994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:18.614996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:18.615521 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:18.615812 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:18.616288 (+ 476us) server_negotiation.cc:299] Negotiation successful
0923 17:33:18.616611 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":346,"threads_started":1}
I20260923 17:33:18.621433 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:18.624168 28764 raft_consensus.cc:348] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638b83033c544bbb91810e1e0116c814" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } }
I20260923 17:33:18.624670 28764 raft_consensus.cc:374] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:18.624778 28764 raft_consensus.cc:729] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638b83033c544bbb91810e1e0116c814, State: Initialized, Role: FOLLOWER
I20260923 17:33:18.625387 28764 consensus_queue.cc:260] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638b83033c544bbb91810e1e0116c814" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } }
I20260923 17:33:18.625627 28764 raft_consensus.cc:388] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:18.625767 28764 raft_consensus.cc:482] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:18.625962 28764 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:18.628611 28764 raft_consensus.cc:504] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638b83033c544bbb91810e1e0116c814" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } }
I20260923 17:33:18.629213 28764 leader_election.cc:302] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 638b83033c544bbb91810e1e0116c814; no voters:
I20260923 17:33:18.629873 28764 leader_election.cc:288] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:18.629976 28774 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:18.632464 28764 sys_catalog.cc:564] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:18.636641 28774 raft_consensus.cc:686] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [term 1 LEADER]: Becoming Leader. State: Replica: 638b83033c544bbb91810e1e0116c814, State: Running, Role: LEADER
I20260923 17:33:18.637372 28774 consensus_queue.cc:237] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638b83033c544bbb91810e1e0116c814" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } }
I20260923 17:33:18.640101 28767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:18.626115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40691 (local address 127.0.0.1:37791)
0923 17:33:18.626315 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:18.626323 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:18.626824 (+ 501us) server_negotiation.cc:407] Connection header received
0923 17:33:18.627000 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:18.627006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:18.627076 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:18.627157 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:18.628645 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:18.629474 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:18.631012 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:18.631531 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:18.631638 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:18.634225 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:18.634232 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:18.634236 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:18.634283 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:18.637147 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:18.637916 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:18.637922 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:18.637925 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:18.638033 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:18.638503 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:18.638508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:18.638511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:18.638980 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:18.639276 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:18.639595 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 17:33:18.639765 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 17:33:18.644171 28776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 638b83033c544bbb91810e1e0116c814. Latest consensus state: current_term: 1 leader_uuid: "638b83033c544bbb91810e1e0116c814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638b83033c544bbb91810e1e0116c814" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } } }
I20260923 17:33:18.644631 28776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:18.645169 28775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "638b83033c544bbb91810e1e0116c814" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "638b83033c544bbb91810e1e0116c814" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37791 } } }
I20260923 17:33:18.645514 28775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:18.645722 28786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:18.651059 28786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:18.660766 28786 catalog_manager.cc:1347] Generated new cluster ID: b4b5c9d3a699472581a9200fd1163967
I20260923 17:33:18.660917 28786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:18.679500 28786 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:18.680864 28786 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:18.692653 28786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 638b83033c544bbb91810e1e0116c814: Generated new TSK 0
I20260923 17:33:18.693784 28786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:18.708307 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 28697
I20260923 17:33:18.709714 28759 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:18.803300 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 28697
W20260923 17:33:18.818053 28682 connection.cc:535] client connection to 127.0.0.1:37791 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37791): Connection reset by peer (error 104)
2026-09-23T17:33:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1951 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:18 dist-test-slave-p7jt krb5kdc[28796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:18 dist-test-slave-p7jt krb5kdc[28796](info): set up 2 sockets
Sep 23 17:33:18 dist-test-slave-p7jt krb5kdc[28796](info): commencing operation
krb5kdc: starting...
W20260923 17:33:19.827037 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.990s user 0.004s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:19 dist-test-slave-p7jt krb5kdc[28796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:20.087193 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38621
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:20.418273 28809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:20.418632 28809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:20.418759 28809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:20.426069 28809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:20.426199 28809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:20.426247 28809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:20.426330 28809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:20.426390 28809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:20.436470 28809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38621
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:20.437882 28809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:20.439383 28809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:20.451328 28814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:20.451366 28815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:20.451284 28817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:20.452373 28809 server_base.cc:1034] running on GCE node
I20260923 17:33:20.453174 28809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:20.454699 28809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:20.455919 28809 hybrid_clock.cc:648] HybridClock initialized: now 1790184800455850 us; error 60 us; skew 500 ppm
I20260923 17:33:20.456265 28809 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:20 dist-test-slave-p7jt krb5kdc[28796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:20.459463 28809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:20.461026 28809 webserver.cc:460] Webserver started at http://127.0.0.1:44745/ using document root <none> and password file <none>
I20260923 17:33:20.461756 28809 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:20.461879 28809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:20.462177 28809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:20.465395 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "e93f2ad2b4e94325b4d20a2eecd6d145"
format_stamp: "Formatted at 2026-09-23 17:33:20 on dist-test-slave-p7jt"
server_key: "f55e98791365dd7909c373a471874342"
server_key_iv: "8726e370add2a4e91ee4c40b59218a69"
server_key_version: "encryptionkey@0"
I20260923 17:33:20.466251 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "e93f2ad2b4e94325b4d20a2eecd6d145"
format_stamp: "Formatted at 2026-09-23 17:33:20 on dist-test-slave-p7jt"
server_key: "f55e98791365dd7909c373a471874342"
server_key_iv: "8726e370add2a4e91ee4c40b59218a69"
server_key_version: "encryptionkey@0"
I20260923 17:33:20.471959 28809 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:33:20.475795 28824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:20.477239 28809 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:20.477434 28809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "e93f2ad2b4e94325b4d20a2eecd6d145"
format_stamp: "Formatted at 2026-09-23 17:33:20 on dist-test-slave-p7jt"
server_key: "f55e98791365dd7909c373a471874342"
server_key_iv: "8726e370add2a4e91ee4c40b59218a69"
server_key_version: "encryptionkey@0"
I20260923 17:33:20.477638 28809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:20.505815 28809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:20.510277 28809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:20.510622 28809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:20.532805 28809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38621
I20260923 17:33:20.532824 28875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38621 every 8 connection(s)
I20260923 17:33:20.534250 28809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:20.537963 28876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:20.541757 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 28809
I20260923 17:33:20.541944 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:20.542397 8311 external_mini_cluster.cc:1428] Setting key df74b253394ff75323e9598e5bad6968
I20260923 17:33:20.550161 28876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145: Bootstrap starting.
I20260923 17:33:20.553438 28876 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:20.554590 28876 log.cc:824] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:20.555517 28879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:20.544220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46723 (local address 127.0.0.1:38621)
0923 17:33:20.545029 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:20.545042 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:20.545085 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:33:20.546781 (+ 1696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:20.546806 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:20.547267 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:20.547664 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:20.548252 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:20.549563 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:20.550985 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:20.551679 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:20.551849 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:20.553130 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:20.553138 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:20.553153 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:20.553728 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:20.553816 (+ 88us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:20.553845 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:20.553982 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:20.554264 (+ 282us) server_negotiation.cc:299] Negotiation successful
0923 17:33:20.554445 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":348,"threads_started":1}
I20260923 17:33:20.557351 28876 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145: No bootstrap required, opened a new log
I20260923 17:33:20.559213 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:20.565500 28876 raft_consensus.cc:348] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93f2ad2b4e94325b4d20a2eecd6d145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38621 } }
I20260923 17:33:20.565958 28876 raft_consensus.cc:374] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:20.566057 28876 raft_consensus.cc:729] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93f2ad2b4e94325b4d20a2eecd6d145, State: Initialized, Role: FOLLOWER
I20260923 17:33:20.566808 28876 consensus_queue.cc:260] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93f2ad2b4e94325b4d20a2eecd6d145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38621 } }
I20260923 17:33:20.567123 28876 raft_consensus.cc:388] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:20.567257 28876 raft_consensus.cc:482] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:20.567425 28876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:20.570422 28879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:20.563353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49235 (local address 127.0.0.1:38621)
0923 17:33:20.563559 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:20.563566 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:20.564040 (+ 474us) server_negotiation.cc:407] Connection header received
0923 17:33:20.564347 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:20.564351 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:20.564426 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:20.564515 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:20.565186 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:20.565962 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:20.567868 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:20.568341 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:20.568458 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:20.569688 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:20.569692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:20.569695 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:20.569714 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:20.569774 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:20.569790 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:20.569896 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:20.570051 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 17:33:20.570098 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:33:20.570384 28876 raft_consensus.cc:504] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93f2ad2b4e94325b4d20a2eecd6d145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38621 } }
I20260923 17:33:20.571023 28876 leader_election.cc:302] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e93f2ad2b4e94325b4d20a2eecd6d145; no voters:
I20260923 17:33:20.571620 28876 leader_election.cc:288] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:20.571765 28886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:20.572482 28886 raft_consensus.cc:686] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [term 1 LEADER]: Becoming Leader. State: Replica: e93f2ad2b4e94325b4d20a2eecd6d145, State: Running, Role: LEADER
I20260923 17:33:20.573036 28886 consensus_queue.cc:237] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93f2ad2b4e94325b4d20a2eecd6d145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38621 } }
I20260923 17:33:20.573720 28876 sys_catalog.cc:564] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:20.575549 28887 sys_catalog.cc:455] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e93f2ad2b4e94325b4d20a2eecd6d145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93f2ad2b4e94325b4d20a2eecd6d145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38621 } } }
I20260923 17:33:20.575649 28888 sys_catalog.cc:455] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e93f2ad2b4e94325b4d20a2eecd6d145. Latest consensus state: current_term: 1 leader_uuid: "e93f2ad2b4e94325b4d20a2eecd6d145" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e93f2ad2b4e94325b4d20a2eecd6d145" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38621 } } }
I20260923 17:33:20.576017 28888 sys_catalog.cc:458] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:20.576017 28887 sys_catalog.cc:458] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:20.577050 28891 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:20.583216 28891 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:20.593986 28891 catalog_manager.cc:1347] Generated new cluster ID: 1461ce8f38a8403b80f00e7001440a55
I20260923 17:33:20.594111 28891 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:20.624558 28891 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:20.625950 28891 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:20.636194 28891 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e93f2ad2b4e94325b4d20a2eecd6d145: Generated new TSK 0
I20260923 17:33:20.637095 28891 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:20.645895 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 28809
W20260923 17:33:20.647110 28871 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46723
W20260923 17:33:20.647915 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 28809: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:20.648094 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 28809
W20260923 17:33:20.662518 28794 connection.cc:535] client connection to 127.0.0.1:38621 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38621): Connection reset by peer (error 104)
2026-09-23T17:33:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1844 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:20 dist-test-slave-p7jt krb5kdc[28907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:20 dist-test-slave-p7jt krb5kdc[28907](info): set up 2 sockets
Sep 23 17:33:20 dist-test-slave-p7jt krb5kdc[28907](info): commencing operation
krb5kdc: starting...
W20260923 17:33:21.682651 8311 mini_kdc.cc:121] Time spent starting KDC: real 1.001s user 0.002s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:21 dist-test-slave-p7jt krb5kdc[28907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184801, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:21.934008 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35601
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:22.266273 28920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:22.266656 28920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:22.266810 28920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:22.274369 28920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:22.274559 28920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:22.274644 28920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:22.274729 28920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:22.274789 28920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:22.285125 28920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35601
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:22.286614 28920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:22.287922 28920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:22.299820 28925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:22.299875 28928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:22.299789 28926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:22.300886 28920 server_base.cc:1034] running on GCE node
I20260923 17:33:22.301669 28920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:22.303239 28920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:22.304450 28920 hybrid_clock.cc:648] HybridClock initialized: now 1790184802304400 us; error 39 us; skew 500 ppm
I20260923 17:33:22.304798 28920 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:22 dist-test-slave-p7jt krb5kdc[28907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:22.308018 28920 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:22.309489 28920 webserver.cc:460] Webserver started at http://127.0.0.1:43867/ using document root <none> and password file <none>
I20260923 17:33:22.310187 28920 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:22.310297 28920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:22.310648 28920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:22.313714 28920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "2714d51ee20d4089861bc60a24a591cc"
format_stamp: "Formatted at 2026-09-23 17:33:22 on dist-test-slave-p7jt"
server_key: "1bc70d4ad17f9c89d8d96f5ea3feab11"
server_key_iv: "0a11855780a3d6e16bcdb3d9b19940b2"
server_key_version: "encryptionkey@0"
I20260923 17:33:22.314535 28920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "2714d51ee20d4089861bc60a24a591cc"
format_stamp: "Formatted at 2026-09-23 17:33:22 on dist-test-slave-p7jt"
server_key: "1bc70d4ad17f9c89d8d96f5ea3feab11"
server_key_iv: "0a11855780a3d6e16bcdb3d9b19940b2"
server_key_version: "encryptionkey@0"
I20260923 17:33:22.319983 28920 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:33:22.323634 28935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:22.324977 28920 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:33:22.325147 28920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "2714d51ee20d4089861bc60a24a591cc"
format_stamp: "Formatted at 2026-09-23 17:33:22 on dist-test-slave-p7jt"
server_key: "1bc70d4ad17f9c89d8d96f5ea3feab11"
server_key_iv: "0a11855780a3d6e16bcdb3d9b19940b2"
server_key_version: "encryptionkey@0"
I20260923 17:33:22.325353 28920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:22.331465 28920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:22.335907 28920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:22.336194 28920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:22.353857 28920 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35601
I20260923 17:33:22.353896 28986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35601 every 8 connection(s)
I20260923 17:33:22.355356 28920 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:22.358661 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 28920
I20260923 17:33:22.358836 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:22.359383 8311 external_mini_cluster.cc:1428] Setting key 31ed2760fb55b6a3f2f3457489d4813b
I20260923 17:33:22.359444 28987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:22.367528 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:22.361162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59503 (local address 127.0.0.1:35601)
0923 17:33:22.361936 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:22.361952 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:22.361996 (+ 44us) server_negotiation.cc:407] Connection header received
0923 17:33:22.364069 (+ 2073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:22.364092 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:22.364454 (+ 362us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:22.364787 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:22.365902 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:22.365912 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:22.365926 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:22.365942 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:22.366010 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:22.366033 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:22.366206 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:22.366308 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:33:22.366532 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":346,"threads_started":1}
I20260923 17:33:22.371572 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:22.373200 28987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc: Bootstrap starting.
I20260923 17:33:22.377209 28987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:22.378645 28987 log.cc:824] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:22.379325 28990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:22.376657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55289 (local address 127.0.0.1:35601)
0923 17:33:22.376809 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:22.376814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:22.377167 (+ 353us) server_negotiation.cc:407] Connection header received
0923 17:33:22.377429 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:22.377434 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:22.377489 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:22.377582 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:22.378581 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:22.378585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:22.378589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:22.378605 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:22.378654 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:22.378675 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:22.378818 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:22.378906 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 17:33:22.378959 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 17:33:22.381829 28987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc: No bootstrap required, opened a new log
I20260923 17:33:22.389298 28987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2714d51ee20d4089861bc60a24a591cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35601 } }
I20260923 17:33:22.389694 28987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:22.389772 28987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2714d51ee20d4089861bc60a24a591cc, State: Initialized, Role: FOLLOWER
I20260923 17:33:22.390370 28987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2714d51ee20d4089861bc60a24a591cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35601 } }
I20260923 17:33:22.390631 28987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:22.390738 28987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:22.390866 28987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:22.393368 28987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2714d51ee20d4089861bc60a24a591cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35601 } }
I20260923 17:33:22.393875 28987 leader_election.cc:302] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2714d51ee20d4089861bc60a24a591cc; no voters:
I20260923 17:33:22.394562 28987 leader_election.cc:288] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:22.394660 28997 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:22.395313 28997 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [term 1 LEADER]: Becoming Leader. State: Replica: 2714d51ee20d4089861bc60a24a591cc, State: Running, Role: LEADER
I20260923 17:33:22.395911 28997 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2714d51ee20d4089861bc60a24a591cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35601 } }
I20260923 17:33:22.396646 28987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:22.398797 28998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2714d51ee20d4089861bc60a24a591cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2714d51ee20d4089861bc60a24a591cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35601 } } }
I20260923 17:33:22.399286 28998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:22.399230 28999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2714d51ee20d4089861bc60a24a591cc. Latest consensus state: current_term: 1 leader_uuid: "2714d51ee20d4089861bc60a24a591cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2714d51ee20d4089861bc60a24a591cc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35601 } } }
I20260923 17:33:22.399519 28999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:22.401450 29006 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:22.406718 29006 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:22.417026 29006 catalog_manager.cc:1347] Generated new cluster ID: e30257ee13dc45469c20fd90ef5e1c0e
I20260923 17:33:22.417172 29006 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:22.446604 29006 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:22.447219 29006 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:22.457716 29006 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2714d51ee20d4089861bc60a24a591cc: Generated new TSK 0
I20260923 17:33:22.458673 29006 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:22.468218 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 28920
W20260923 17:33:22.469887 28982 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59503
W20260923 17:33:22.470707 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 28920: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:22.470852 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 28920
2026-09-23T17:33:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1822 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:22 dist-test-slave-p7jt krb5kdc[29018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:22 dist-test-slave-p7jt krb5kdc[29018](info): set up 2 sockets
Sep 23 17:33:22 dist-test-slave-p7jt krb5kdc[29018](info): commencing operation
krb5kdc: starting...
W20260923 17:33:23.502723 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:23 dist-test-slave-p7jt krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:23.754014 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36991
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34859
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36991
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:24.084705 29031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:24.085027 29031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:24.085165 29031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:24.092782 29031 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:24.092926 29031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:24.093012 29031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:24.093096 29031 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:24.093159 29031 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:24.103331 29031 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34859
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:24.104664 29031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:24.105971 29031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:24.117808 29037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:24.117851 29039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:24.117815 29036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:24.118590 29031 server_base.cc:1034] running on GCE node
I20260923 17:33:24.119457 29031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:24.120968 29031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:24.122221 29031 hybrid_clock.cc:648] HybridClock initialized: now 1790184804122168 us; error 34 us; skew 500 ppm
I20260923 17:33:24.122705 29031 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:24 dist-test-slave-p7jt krb5kdc[29018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:24.149418 29031 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:24.151333 29031 webserver.cc:460] Webserver started at http://127.0.0.1:41213/ using document root <none> and password file <none>
I20260923 17:33:24.152161 29031 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:24.152303 29031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:24.152678 29031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:24.156145 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "c680d480591e42d589a025c253f0fc1c"
format_stamp: "Formatted at 2026-09-23 17:33:24 on dist-test-slave-p7jt"
server_key: "039368e6f1855d2946685c394ecf4fe2"
server_key_iv: "e1f1f0f7a8fefe9e193443b8211bc926"
server_key_version: "encryptionkey@0"
I20260923 17:33:24.156975 29031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "c680d480591e42d589a025c253f0fc1c"
format_stamp: "Formatted at 2026-09-23 17:33:24 on dist-test-slave-p7jt"
server_key: "039368e6f1855d2946685c394ecf4fe2"
server_key_iv: "e1f1f0f7a8fefe9e193443b8211bc926"
server_key_version: "encryptionkey@0"
I20260923 17:33:24.162537 29031 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:33:24.166144 29046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:24.167661 29031 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:24.167894 29031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "c680d480591e42d589a025c253f0fc1c"
format_stamp: "Formatted at 2026-09-23 17:33:24 on dist-test-slave-p7jt"
server_key: "039368e6f1855d2946685c394ecf4fe2"
server_key_iv: "e1f1f0f7a8fefe9e193443b8211bc926"
server_key_version: "encryptionkey@0"
I20260923 17:33:24.168105 29031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:24.189615 29031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:24.194061 29031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:24.194353 29031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:24.212451 29031 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36991
I20260923 17:33:24.212471 29097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36991 every 8 connection(s)
I20260923 17:33:24.213822 29031 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:24.217612 29098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:24.218869 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 29031
I20260923 17:33:24.219044 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:24.219470 8311 external_mini_cluster.cc:1428] Setting key 29b942ccdbaf77036c42761364e565c8
I20260923 17:33:24.232268 29098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c: Bootstrap starting.
Sep 23 17:33:24 dist-test-slave-p7jt krb5kdc[29018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184803, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:33:24.236385 29098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:24.237845 29098 log.cc:824] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:24.241526 29098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c: No bootstrap required, opened a new log
I20260923 17:33:24.245692 29101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:24.221280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48741 (local address 127.0.0.1:36991)
0923 17:33:24.222095 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:24.222108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:24.222153 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:33:24.223854 (+ 1701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:24.223880 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:24.224488 (+ 608us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:24.224894 (+ 406us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:24.226821 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:24.228231 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:24.230179 (+ 1948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:24.231109 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:24.231289 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:24.236612 (+ 5323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:24.236626 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:24.236649 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:24.236711 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:24.240572 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:24.241418 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:24.241427 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:24.241436 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:24.241671 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:24.242121 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:24.242130 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:24.242133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:24.242924 (+ 791us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:24.243232 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:24.243732 (+ 500us) server_negotiation.cc:299] Negotiation successful
0923 17:33:24.244238 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":376,"threads_started":1}
I20260923 17:33:24.250221 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:24.251289 29098 raft_consensus.cc:348] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680d480591e42d589a025c253f0fc1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } }
I20260923 17:33:24.251739 29098 raft_consensus.cc:374] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:24.251837 29098 raft_consensus.cc:729] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c680d480591e42d589a025c253f0fc1c, State: Initialized, Role: FOLLOWER
I20260923 17:33:24.252427 29098 consensus_queue.cc:260] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680d480591e42d589a025c253f0fc1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } }
I20260923 17:33:24.252647 29098 raft_consensus.cc:388] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:24.252771 29098 raft_consensus.cc:482] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:24.252928 29098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:24.255437 29098 raft_consensus.cc:504] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680d480591e42d589a025c253f0fc1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } }
I20260923 17:33:24.256023 29098 leader_election.cc:302] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c680d480591e42d589a025c253f0fc1c; no voters:
I20260923 17:33:24.256614 29098 leader_election.cc:288] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:24.256776 29108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:24.257774 29108 raft_consensus.cc:686] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [term 1 LEADER]: Becoming Leader. State: Replica: c680d480591e42d589a025c253f0fc1c, State: Running, Role: LEADER
I20260923 17:33:24.258355 29108 consensus_queue.cc:237] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680d480591e42d589a025c253f0fc1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } }
I20260923 17:33:24.258877 29098 sys_catalog.cc:564] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:24.263216 29109 sys_catalog.cc:455] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c680d480591e42d589a025c253f0fc1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680d480591e42d589a025c253f0fc1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } } }
I20260923 17:33:24.263620 29109 sys_catalog.cc:458] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:24.271526 29110 sys_catalog.cc:455] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c680d480591e42d589a025c253f0fc1c. Latest consensus state: current_term: 1 leader_uuid: "c680d480591e42d589a025c253f0fc1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c680d480591e42d589a025c253f0fc1c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36991 } } }
I20260923 17:33:24.271889 29110 sys_catalog.cc:458] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:24.272440 29101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:24.254832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60421 (local address 127.0.0.1:36991)
0923 17:33:24.255029 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:24.255036 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:24.255385 (+ 349us) server_negotiation.cc:407] Connection header received
0923 17:33:24.255532 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:24.255538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:24.255618 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:24.255718 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:24.257527 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:24.258636 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:24.260442 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:24.261215 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:24.261343 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:24.264487 (+ 3144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:24.264494 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:24.264500 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:24.264538 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:24.268448 (+ 3910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:24.269421 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:24.269428 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:24.269433 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:24.269613 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:24.270110 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:24.270115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:24.270119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:24.270594 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:24.270844 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:24.271766 (+ 922us) server_negotiation.cc:299] Negotiation successful
0923 17:33:24.272057 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:33:24.273087 29117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:24.278090 29117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:24.288249 29117 catalog_manager.cc:1347] Generated new cluster ID: 54b3834ebc474c71ba9de40280a47b73
I20260923 17:33:24.288391 29117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:24.300122 29117 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:24.301836 29117 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:24.307978 29117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c680d480591e42d589a025c253f0fc1c: Generated new TSK 0
I20260923 17:33:24.309073 29117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:24.321655 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29031
I20260923 17:33:24.323227 29093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:24.414834 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 29031
2026-09-23T17:33:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1944 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:24 dist-test-slave-p7jt krb5kdc[29130](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:24 dist-test-slave-p7jt krb5kdc[29130](info): set up 2 sockets
Sep 23 17:33:24 dist-test-slave-p7jt krb5kdc[29130](info): commencing operation
krb5kdc: starting...
W20260923 17:33:25.446691 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.998s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:25 dist-test-slave-p7jt krb5kdc[29130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184805, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:25.705523 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:26.032527 29143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:26.032850 29143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:26.032943 29143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:26.040343 29143 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:26.040467 29143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:26.040512 29143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:26.040570 29143 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:26.040613 29143 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:26.051049 29143 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:26.052309 29143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:26.053630 29143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:26.065349 29148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:26.065483 29149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:26.065330 29151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:26.066010 29143 server_base.cc:1034] running on GCE node
I20260923 17:33:26.066870 29143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:26.068400 29143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:26.069651 29143 hybrid_clock.cc:648] HybridClock initialized: now 1790184806069584 us; error 57 us; skew 500 ppm
I20260923 17:33:26.070057 29143 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:26 dist-test-slave-p7jt krb5kdc[29130](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:26.073443 29143 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:26.075281 29143 webserver.cc:460] Webserver started at http://127.0.0.1:44085/ using document root <none> and password file <none>
I20260923 17:33:26.076018 29143 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:26.076159 29143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:26.076493 29143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:26.079797 29143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "f41fe4a6e23848fd81e1cf6d3f877340"
format_stamp: "Formatted at 2026-09-23 17:33:26 on dist-test-slave-p7jt"
server_key: "586c6fdea2fcf57421fe0f64a28ae58a"
server_key_iv: "633bcfd578d20d11838c486d0984fdb7"
server_key_version: "encryptionkey@0"
I20260923 17:33:26.080626 29143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "f41fe4a6e23848fd81e1cf6d3f877340"
format_stamp: "Formatted at 2026-09-23 17:33:26 on dist-test-slave-p7jt"
server_key: "586c6fdea2fcf57421fe0f64a28ae58a"
server_key_iv: "633bcfd578d20d11838c486d0984fdb7"
server_key_version: "encryptionkey@0"
I20260923 17:33:26.086344 29143 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260923 17:33:26.090150 29158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:26.091631 29143 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:26.091807 29143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "f41fe4a6e23848fd81e1cf6d3f877340"
format_stamp: "Formatted at 2026-09-23 17:33:26 on dist-test-slave-p7jt"
server_key: "586c6fdea2fcf57421fe0f64a28ae58a"
server_key_iv: "633bcfd578d20d11838c486d0984fdb7"
server_key_version: "encryptionkey@0"
I20260923 17:33:26.092010 29143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:26.103255 29143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:26.107448 29143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:26.107744 29143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:26.125002 29143 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34009
I20260923 17:33:26.125036 29209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34009 every 8 connection(s)
I20260923 17:33:26.126401 29143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:26.129801 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 29143
I20260923 17:33:26.129980 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:26.130427 8311 external_mini_cluster.cc:1428] Setting key 724645f488d6df5e0bd4254e88a0cfa0
I20260923 17:33:26.130508 29210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:26.145776 29210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340: Bootstrap starting.
I20260923 17:33:26.145925 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:26.132282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55329 (local address 127.0.0.1:34009)
0923 17:33:26.133118 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:26.133134 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:26.133197 (+ 63us) server_negotiation.cc:407] Connection header received
0923 17:33:26.135423 (+ 2226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:26.135454 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:26.136020 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:26.136496 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:26.137050 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:26.139101 (+ 2051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:26.140517 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:26.141559 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:26.141748 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:26.142916 (+ 1168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:26.142930 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:26.142954 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:26.143725 (+ 771us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:26.143836 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:26.143873 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:26.143985 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:26.144260 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 17:33:26.144556 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":347,"threads_started":1}
I20260923 17:33:26.149242 29210 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:26.149813 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:26.150333 29210 log.cc:824] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:26.153236 29210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340: No bootstrap required, opened a new log
I20260923 17:33:26.160200 29213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:26.154341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39619 (local address 127.0.0.1:34009)
0923 17:33:26.154535 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:26.154542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:26.154926 (+ 384us) server_negotiation.cc:407] Connection header received
0923 17:33:26.155109 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:26.155112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:26.155179 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:26.155263 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:26.155766 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:26.156558 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:26.157879 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:26.158348 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:26.158488 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:26.159501 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:26.159506 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:26.159509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:26.159525 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:26.159564 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:26.159580 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:26.159658 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:26.159810 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 17:33:26.159902 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260923 17:33:26.161284 29210 raft_consensus.cc:348] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41fe4a6e23848fd81e1cf6d3f877340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260923 17:33:26.161746 29210 raft_consensus.cc:374] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:26.161839 29210 raft_consensus.cc:729] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f41fe4a6e23848fd81e1cf6d3f877340, State: Initialized, Role: FOLLOWER
I20260923 17:33:26.162389 29210 consensus_queue.cc:260] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41fe4a6e23848fd81e1cf6d3f877340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260923 17:33:26.162650 29210 raft_consensus.cc:388] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:26.162788 29210 raft_consensus.cc:482] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:26.162936 29210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:26.165112 29210 raft_consensus.cc:504] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41fe4a6e23848fd81e1cf6d3f877340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260923 17:33:26.165598 29210 leader_election.cc:302] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f41fe4a6e23848fd81e1cf6d3f877340; no voters:
I20260923 17:33:26.166271 29210 leader_election.cc:288] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:26.166450 29220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:26.167421 29220 raft_consensus.cc:686] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [term 1 LEADER]: Becoming Leader. State: Replica: f41fe4a6e23848fd81e1cf6d3f877340, State: Running, Role: LEADER
I20260923 17:33:26.168015 29220 consensus_queue.cc:237] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41fe4a6e23848fd81e1cf6d3f877340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260923 17:33:26.168637 29210 sys_catalog.cc:564] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:26.170931 29221 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f41fe4a6e23848fd81e1cf6d3f877340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41fe4a6e23848fd81e1cf6d3f877340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260923 17:33:26.171334 29221 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:26.172959 29228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:26.171851 29222 sys_catalog.cc:455] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f41fe4a6e23848fd81e1cf6d3f877340. Latest consensus state: current_term: 1 leader_uuid: "f41fe4a6e23848fd81e1cf6d3f877340" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f41fe4a6e23848fd81e1cf6d3f877340" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260923 17:33:26.173249 29222 sys_catalog.cc:458] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:26.179090 29228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:26.188606 29228 catalog_manager.cc:1347] Generated new cluster ID: 0e81c2a242274609ae2565d254da0851
I20260923 17:33:26.188791 29228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:26.206434 29228 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:26.208366 29228 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:26.227082 29228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f41fe4a6e23848fd81e1cf6d3f877340: Generated new TSK 0
I20260923 17:33:26.228128 29228 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:26.246670 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29143
W20260923 17:33:26.248204 29205 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:55329
W20260923 17:33:26.249030 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29143: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:26.249218 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 29143
2026-09-23T17:33:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:26 dist-test-slave-p7jt krb5kdc[29241](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:26 dist-test-slave-p7jt krb5kdc[29241](info): set up 2 sockets
Sep 23 17:33:26 dist-test-slave-p7jt krb5kdc[29241](info): commencing operation
krb5kdc: starting...
W20260923 17:33:27.270680 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.988s user 0.004s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:27 dist-test-slave-p7jt krb5kdc[29241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184807, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:27.523744 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46057
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:27.852826 29254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:27.853124 29254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:27.853250 29254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:27.860529 29254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:27.860643 29254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:27.860719 29254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:27.860801 29254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:27.860862 29254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:27.870822 29254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46057
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:27.872148 29254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:27.873484 29254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:27.884913 29262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:27.885001 29260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:27.885020 29259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:27.886032 29254 server_base.cc:1034] running on GCE node
I20260923 17:33:27.886842 29254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:27.888303 29254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:27.889515 29254 hybrid_clock.cc:648] HybridClock initialized: now 1790184807889469 us; error 52 us; skew 500 ppm
I20260923 17:33:27.889884 29254 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:27 dist-test-slave-p7jt krb5kdc[29241](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184807, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:27.892886 29254 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:27.894289 29254 webserver.cc:460] Webserver started at http://127.0.0.1:34377/ using document root <none> and password file <none>
I20260923 17:33:27.894987 29254 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:27.895084 29254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:27.895334 29254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:27.898211 29254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "5cb06b4220464bc7aefa2096066e839a"
format_stamp: "Formatted at 2026-09-23 17:33:27 on dist-test-slave-p7jt"
server_key: "19aa4748fc132ef55361e2f5c490b816"
server_key_iv: "fb408b0b29e14122e0dad5cd10019887"
server_key_version: "encryptionkey@0"
I20260923 17:33:27.898995 29254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "5cb06b4220464bc7aefa2096066e839a"
format_stamp: "Formatted at 2026-09-23 17:33:27 on dist-test-slave-p7jt"
server_key: "19aa4748fc132ef55361e2f5c490b816"
server_key_iv: "fb408b0b29e14122e0dad5cd10019887"
server_key_version: "encryptionkey@0"
I20260923 17:33:27.903913 29254 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 17:33:27.907287 29269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:27.908638 29254 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:27.908829 29254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "5cb06b4220464bc7aefa2096066e839a"
format_stamp: "Formatted at 2026-09-23 17:33:27 on dist-test-slave-p7jt"
server_key: "19aa4748fc132ef55361e2f5c490b816"
server_key_iv: "fb408b0b29e14122e0dad5cd10019887"
server_key_version: "encryptionkey@0"
I20260923 17:33:27.909010 29254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:27.914515 29254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:27.918496 29254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:27.918781 29254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:27.935850 29254 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46057
I20260923 17:33:27.935900 29320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46057 every 8 connection(s)
I20260923 17:33:27.937250 29254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:27.937856 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 29254
I20260923 17:33:27.938011 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:27.938588 8311 external_mini_cluster.cc:1428] Setting key 33806d62d63904df794bc8dfeeba923c
I20260923 17:33:27.941542 29321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:27.946704 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:27.940382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48537 (local address 127.0.0.1:46057)
0923 17:33:27.941140 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:27.941152 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:27.941193 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:33:27.943162 (+ 1969us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:27.943196 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:27.943574 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:27.943903 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:27.944970 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:27.944977 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:27.944994 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:27.945018 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:27.945096 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:27.945119 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:27.945310 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:27.945452 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 17:33:27.945648 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":324,"threads_started":1}
I20260923 17:33:27.950729 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:27.953917 29321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a: Bootstrap starting.
I20260923 17:33:27.957031 29321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:27.957451 29324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:27.954968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44019 (local address 127.0.0.1:46057)
0923 17:33:27.955164 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:27.955172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:27.955631 (+ 459us) server_negotiation.cc:407] Connection header received
0923 17:33:27.955812 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:27.955817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:27.955871 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:27.955949 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:27.956814 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:27.956820 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:27.956824 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:27.956840 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:27.956880 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:27.956895 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:27.956996 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:27.957094 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 17:33:27.957158 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:33:27.958302 29321 log.cc:824] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:27.961089 29321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a: No bootstrap required, opened a new log
I20260923 17:33:27.968535 29321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb06b4220464bc7aefa2096066e839a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260923 17:33:27.968925 29321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:27.969000 29321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb06b4220464bc7aefa2096066e839a, State: Initialized, Role: FOLLOWER
I20260923 17:33:27.969537 29321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb06b4220464bc7aefa2096066e839a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260923 17:33:27.969739 29321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:27.969874 29321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:27.970045 29321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:27.972558 29321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb06b4220464bc7aefa2096066e839a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260923 17:33:27.973049 29321 leader_election.cc:302] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cb06b4220464bc7aefa2096066e839a; no voters:
I20260923 17:33:27.973634 29321 leader_election.cc:288] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:27.973758 29331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:27.974479 29331 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [term 1 LEADER]: Becoming Leader. State: Replica: 5cb06b4220464bc7aefa2096066e839a, State: Running, Role: LEADER
I20260923 17:33:27.974996 29331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb06b4220464bc7aefa2096066e839a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } }
I20260923 17:33:27.975678 29321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:27.977720 29332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cb06b4220464bc7aefa2096066e839a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb06b4220464bc7aefa2096066e839a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } } }
I20260923 17:33:27.978138 29332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:27.978892 29333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cb06b4220464bc7aefa2096066e839a. Latest consensus state: current_term: 1 leader_uuid: "5cb06b4220464bc7aefa2096066e839a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cb06b4220464bc7aefa2096066e839a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46057 } } }
I20260923 17:33:27.979177 29333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:27.981179 29336 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:27.987380 29336 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:27.997679 29336 catalog_manager.cc:1347] Generated new cluster ID: f63e10dac4204aa781c6cd7c0ced9fb9
I20260923 17:33:27.997808 29336 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:28.007051 29336 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:28.007498 29336 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:28.021557 29336 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cb06b4220464bc7aefa2096066e839a: Generated new TSK 0
I20260923 17:33:28.022536 29336 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:28.028924 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29254
W20260923 17:33:28.030005 29316 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48537
W20260923 17:33:28.030725 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29254: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:28.030857 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 29254
2026-09-23T17:33:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1781 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:28 dist-test-slave-p7jt krb5kdc[29352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:28 dist-test-slave-p7jt krb5kdc[29352](info): set up 2 sockets
Sep 23 17:33:28 dist-test-slave-p7jt krb5kdc[29352](info): commencing operation
krb5kdc: starting...
W20260923 17:33:29.070736 8311 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:29 dist-test-slave-p7jt krb5kdc[29352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:29.334617 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35711
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:29.673026 29365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:29.673349 29365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:29.673496 29365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:29.680996 29365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:29.681124 29365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:29.681203 29365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:29.681282 29365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:29.681339 29365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:29.692567 29365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35711
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:29.693894 29365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:29.695237 29365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:29.706878 29370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:29.706950 29373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:29.706975 29371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:29.708068 29365 server_base.cc:1034] running on GCE node
I20260923 17:33:29.708901 29365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:29.710686 29365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:29.711921 29365 hybrid_clock.cc:648] HybridClock initialized: now 1790184809711878 us; error 64 us; skew 500 ppm
I20260923 17:33:29.712334 29365 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:29 dist-test-slave-p7jt krb5kdc[29352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:29.719591 29365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:29.721304 29365 webserver.cc:460] Webserver started at http://127.0.0.1:43265/ using document root <none> and password file <none>
I20260923 17:33:29.722079 29365 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:29.722195 29365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:29.722540 29365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:29.725962 29365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "3bd1d50109be4c8981e93cc71a8f7cef"
format_stamp: "Formatted at 2026-09-23 17:33:29 on dist-test-slave-p7jt"
server_key: "2e165f49cb42781ed019e8395d0179cf"
server_key_iv: "a995c06744f220a6c186149c43695b38"
server_key_version: "encryptionkey@0"
I20260923 17:33:29.726861 29365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "3bd1d50109be4c8981e93cc71a8f7cef"
format_stamp: "Formatted at 2026-09-23 17:33:29 on dist-test-slave-p7jt"
server_key: "2e165f49cb42781ed019e8395d0179cf"
server_key_iv: "a995c06744f220a6c186149c43695b38"
server_key_version: "encryptionkey@0"
I20260923 17:33:29.732422 29365 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:33:29.736063 29380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:29.737579 29365 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:33:29.737773 29365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "3bd1d50109be4c8981e93cc71a8f7cef"
format_stamp: "Formatted at 2026-09-23 17:33:29 on dist-test-slave-p7jt"
server_key: "2e165f49cb42781ed019e8395d0179cf"
server_key_iv: "a995c06744f220a6c186149c43695b38"
server_key_version: "encryptionkey@0"
I20260923 17:33:29.737986 29365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:29.761714 29365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:29.766417 29365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:29.766765 29365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:29.786448 29365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35711
I20260923 17:33:29.786501 29433 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35711 every 8 connection(s)
I20260923 17:33:29.787904 29365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:29.789875 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 29365
I20260923 17:33:29.790043 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:29.790503 8311 external_mini_cluster.cc:1428] Setting key 043c7563e1685234fa33c213772b53e5
I20260923 17:33:29.793592 29434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:33:29 dist-test-slave-p7jt krb5kdc[29352](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184809, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:33:29.805032 29434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef: Bootstrap starting.
I20260923 17:33:29.808429 29434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:29.809517 29434 log.cc:824] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:29.812638 29434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef: No bootstrap required, opened a new log
I20260923 17:33:29.812697 29438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:29.792133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:54139 (local address 127.0.0.1:35711)
0923 17:33:29.792848 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:29.792865 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:29.792912 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:33:29.794689 (+ 1777us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:29.794714 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:29.795158 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:29.795529 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:29.796995 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:29.798490 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:29.799780 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:29.800463 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:29.800632 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:29.805185 (+ 4553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:29.805197 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:29.805220 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:29.805269 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:29.808275 (+ 3006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:29.809168 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:29.809175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:29.809182 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:29.809382 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:29.809851 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:29.809855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:29.809857 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:29.810449 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:29.810823 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:29.811327 (+ 504us) server_negotiation.cc:299] Negotiation successful
0923 17:33:29.811656 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":298,"threads_started":1}
I20260923 17:33:29.816923 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:29.821208 29434 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd1d50109be4c8981e93cc71a8f7cef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260923 17:33:29.821764 29434 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:29.821878 29434 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bd1d50109be4c8981e93cc71a8f7cef, State: Initialized, Role: FOLLOWER
I20260923 17:33:29.822561 29434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd1d50109be4c8981e93cc71a8f7cef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260923 17:33:29.822901 29434 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:29.823051 29434 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:29.823228 29434 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:29.825572 29434 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd1d50109be4c8981e93cc71a8f7cef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260923 17:33:29.826114 29434 leader_election.cc:302] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3bd1d50109be4c8981e93cc71a8f7cef; no voters:
I20260923 17:33:29.826958 29434 leader_election.cc:288] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:29.827042 29445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:29.830382 29445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [term 1 LEADER]: Becoming Leader. State: Replica: 3bd1d50109be4c8981e93cc71a8f7cef, State: Running, Role: LEADER
I20260923 17:33:29.831040 29445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd1d50109be4c8981e93cc71a8f7cef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } }
I20260923 17:33:29.831218 29434 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:29.835037 29447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3bd1d50109be4c8981e93cc71a8f7cef. Latest consensus state: current_term: 1 leader_uuid: "3bd1d50109be4c8981e93cc71a8f7cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd1d50109be4c8981e93cc71a8f7cef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } } }
I20260923 17:33:29.835601 29447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:29.836949 29438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:29.821373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:59905 (local address 127.0.0.1:35711)
0923 17:33:29.821547 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:29.821554 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:29.822051 (+ 497us) server_negotiation.cc:407] Connection header received
0923 17:33:29.822249 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:29.822255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:29.822330 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:29.822413 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:29.823945 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:29.824774 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:29.826068 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:29.826598 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:29.826715 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:29.829155 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:29.829162 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:29.829167 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:29.829213 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:29.832345 (+ 3132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:29.833340 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:29.833350 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:29.833353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:29.833576 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:29.834159 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:29.834177 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:29.834181 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:29.834649 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:29.834990 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:29.836332 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0923 17:33:29.836540 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260923 17:33:29.837239 29453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:29.836571 29446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3bd1d50109be4c8981e93cc71a8f7cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3bd1d50109be4c8981e93cc71a8f7cef" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35711 } } }
I20260923 17:33:29.837730 29446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:29.843632 29453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:29.853961 29453 catalog_manager.cc:1347] Generated new cluster ID: 446f8fea03384505bf56b756cad0f4a2
I20260923 17:33:29.854102 29453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:29.880175 29453 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:29.881775 29453 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:29.888671 29453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3bd1d50109be4c8981e93cc71a8f7cef: Generated new TSK 0
I20260923 17:33:29.889572 29453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:29.909017 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29365
I20260923 17:33:29.910418 29429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:30.006184 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 29365
W20260923 17:33:30.020929 29350 connection.cc:535] client connection to 127.0.0.1:35711 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35711): Connection reset by peer (error 104)
2026-09-23T17:33:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1975 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:30 dist-test-slave-p7jt krb5kdc[29467](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:30 dist-test-slave-p7jt krb5kdc[29467](info): set up 2 sockets
Sep 23 17:33:30 dist-test-slave-p7jt krb5kdc[29467](info): commencing operation
krb5kdc: starting...
W20260923 17:33:31.039181 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.999s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:31 dist-test-slave-p7jt krb5kdc[29467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:31.294171 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45019
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38799
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:31.627406 29480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:31.627756 29480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:31.627907 29480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:31.635417 29480 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:31.635565 29480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:31.635637 29480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:31.635728 29480 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:31.635790 29480 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:31.646219 29480 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38799
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:31.647607 29480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:31.648964 29480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:31.660868 29485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:31.660868 29486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:31.660874 29488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:31.661791 29480 server_base.cc:1034] running on GCE node
I20260923 17:33:31.662585 29480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:31.664072 29480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:31.665297 29480 hybrid_clock.cc:648] HybridClock initialized: now 1790184811665237 us; error 50 us; skew 500 ppm
I20260923 17:33:31.665702 29480 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:31 dist-test-slave-p7jt krb5kdc[29467](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:31.668987 29480 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:31.670542 29480 webserver.cc:460] Webserver started at http://127.0.0.1:41781/ using document root <none> and password file <none>
I20260923 17:33:31.671217 29480 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:31.671317 29480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:31.671561 29480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:31.674566 29480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "7768e5b96c2641ca8f07d75453eef31e"
format_stamp: "Formatted at 2026-09-23 17:33:31 on dist-test-slave-p7jt"
server_key: "71a5fd3eeca32ffe8cb2af63c462ada4"
server_key_iv: "619333f54614fbdbd9c1acd26b741fb6"
server_key_version: "encryptionkey@0"
I20260923 17:33:31.675400 29480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "7768e5b96c2641ca8f07d75453eef31e"
format_stamp: "Formatted at 2026-09-23 17:33:31 on dist-test-slave-p7jt"
server_key: "71a5fd3eeca32ffe8cb2af63c462ada4"
server_key_iv: "619333f54614fbdbd9c1acd26b741fb6"
server_key_version: "encryptionkey@0"
I20260923 17:33:31.680670 29480 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260923 17:33:31.684195 29495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:31.685683 29480 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 17:33:31.685845 29480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "7768e5b96c2641ca8f07d75453eef31e"
format_stamp: "Formatted at 2026-09-23 17:33:31 on dist-test-slave-p7jt"
server_key: "71a5fd3eeca32ffe8cb2af63c462ada4"
server_key_iv: "619333f54614fbdbd9c1acd26b741fb6"
server_key_version: "encryptionkey@0"
I20260923 17:33:31.686033 29480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:31.711761 29480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:31.716596 29480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:31.716877 29480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:31.735675 29480 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45019
I20260923 17:33:31.735697 29546 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45019 every 8 connection(s)
I20260923 17:33:31.737066 29480 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:31.739404 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 29480
I20260923 17:33:31.739591 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:31.740062 8311 external_mini_cluster.cc:1428] Setting key 5b8fd714c68905d4a6988549ee48878e
I20260923 17:33:31.741173 29547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:31.755035 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:31.741809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:49185 (local address 127.0.0.1:45019)
0923 17:33:31.742621 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:31.742636 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:31.742687 (+ 51us) server_negotiation.cc:407] Connection header received
0923 17:33:31.744604 (+ 1917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:31.744635 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:31.745224 (+ 589us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:31.745700 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:31.746231 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:31.748289 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:31.749622 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:31.750688 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:31.750884 (+ 196us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:31.752037 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:31.752047 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:31.752072 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:31.752844 (+ 772us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:31.752939 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:31.752980 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:31.753134 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:31.753425 (+ 291us) server_negotiation.cc:299] Negotiation successful
0923 17:33:31.753633 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":311,"threads_started":1}
I20260923 17:33:31.756003 29547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e: Bootstrap starting.
I20260923 17:33:31.758960 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:31.759244 29547 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:31.760459 29547 log.cc:824] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:31.763975 29547 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e: No bootstrap required, opened a new log
I20260923 17:33:31.771553 29550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:31.764262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:47693 (local address 127.0.0.1:45019)
0923 17:33:31.764436 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:31.764442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:31.765024 (+ 582us) server_negotiation.cc:407] Connection header received
0923 17:33:31.765261 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:31.765266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:31.765339 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:31.765426 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:31.766108 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:31.767030 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:31.768884 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:31.769410 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:31.769532 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:31.770796 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:31.770801 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:31.770805 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:31.770831 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:31.770872 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:31.770890 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:31.770980 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:31.771177 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 17:33:31.771227 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 17:33:31.773857 29547 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7768e5b96c2641ca8f07d75453eef31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45019 } }
I20260923 17:33:31.774250 29547 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:31.774355 29547 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7768e5b96c2641ca8f07d75453eef31e, State: Initialized, Role: FOLLOWER
I20260923 17:33:31.775008 29547 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7768e5b96c2641ca8f07d75453eef31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45019 } }
I20260923 17:33:31.775234 29547 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:31.775377 29547 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:31.775529 29547 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:31.777956 29547 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7768e5b96c2641ca8f07d75453eef31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45019 } }
I20260923 17:33:31.778501 29547 leader_election.cc:302] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7768e5b96c2641ca8f07d75453eef31e; no voters:
I20260923 17:33:31.779132 29547 leader_election.cc:288] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:31.779247 29557 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:31.781203 29547 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:31.782850 29557 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [term 1 LEADER]: Becoming Leader. State: Replica: 7768e5b96c2641ca8f07d75453eef31e, State: Running, Role: LEADER
I20260923 17:33:31.783414 29557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7768e5b96c2641ca8f07d75453eef31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45019 } }
I20260923 17:33:31.786305 29558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7768e5b96c2641ca8f07d75453eef31e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7768e5b96c2641ca8f07d75453eef31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45019 } } }
I20260923 17:33:31.786754 29558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:31.786278 29559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7768e5b96c2641ca8f07d75453eef31e. Latest consensus state: current_term: 1 leader_uuid: "7768e5b96c2641ca8f07d75453eef31e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7768e5b96c2641ca8f07d75453eef31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45019 } } }
I20260923 17:33:31.790686 29559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:31.795463 29567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:31.799763 29567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:31.808930 29567 catalog_manager.cc:1347] Generated new cluster ID: f1d8cafdd3074bfbac5790a4aa27851d
I20260923 17:33:31.809038 29567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:31.820125 29567 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:31.821449 29567 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:31.830732 29567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7768e5b96c2641ca8f07d75453eef31e: Generated new TSK 0
I20260923 17:33:31.831599 29567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:31.840276 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29480
W20260923 17:33:31.841394 29542 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:49185
W20260923 17:33:31.842129 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29480: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:31.842270 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 29480
W20260923 17:33:31.856108 29465 connection.cc:535] client connection to 127.0.0.1:45019 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45019): Connection reset by peer (error 104)
2026-09-23T17:33:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1834 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:31 dist-test-slave-p7jt krb5kdc[29578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:31 dist-test-slave-p7jt krb5kdc[29578](info): set up 2 sockets
Sep 23 17:33:31 dist-test-slave-p7jt krb5kdc[29578](info): commencing operation
krb5kdc: starting...
W20260923 17:33:32.866775 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.992s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:33 dist-test-slave-p7jt krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:33.126106 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38439
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38995
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:33.468231 29591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:33.468556 29591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:33.468681 29591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:33.476217 29591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:33.476359 29591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:33.476441 29591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:33.476526 29591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:33.476584 29591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:33.486654 29591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38995
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:33.487986 29591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:33.489351 29591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:33.501219 29599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:33.501297 29596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:33.501268 29597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:33.502173 29591 server_base.cc:1034] running on GCE node
I20260923 17:33:33.503036 29591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:33.504482 29591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:33.505715 29591 hybrid_clock.cc:648] HybridClock initialized: now 1790184813505655 us; error 49 us; skew 500 ppm
I20260923 17:33:33.506072 29591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:33 dist-test-slave-p7jt krb5kdc[29578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:33.509203 29591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:33.510787 29591 webserver.cc:460] Webserver started at http://127.0.0.1:44785/ using document root <none> and password file <none>
I20260923 17:33:33.511461 29591 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:33.511561 29591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:33.511828 29591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:33.514997 29591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "2d9fbfa6c6584e49a1fe7e6324add29f"
format_stamp: "Formatted at 2026-09-23 17:33:33 on dist-test-slave-p7jt"
server_key: "4ea8a447f1a9c1dbc5138ba887a0a6db"
server_key_iv: "082aad3372b8972fe16d8840ef33a949"
server_key_version: "encryptionkey@0"
I20260923 17:33:33.515745 29591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "2d9fbfa6c6584e49a1fe7e6324add29f"
format_stamp: "Formatted at 2026-09-23 17:33:33 on dist-test-slave-p7jt"
server_key: "4ea8a447f1a9c1dbc5138ba887a0a6db"
server_key_iv: "082aad3372b8972fe16d8840ef33a949"
server_key_version: "encryptionkey@0"
I20260923 17:33:33.521066 29591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260923 17:33:33.524876 29606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:33.526296 29591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:33:33.526448 29591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "2d9fbfa6c6584e49a1fe7e6324add29f"
format_stamp: "Formatted at 2026-09-23 17:33:33 on dist-test-slave-p7jt"
server_key: "4ea8a447f1a9c1dbc5138ba887a0a6db"
server_key_iv: "082aad3372b8972fe16d8840ef33a949"
server_key_version: "encryptionkey@0"
I20260923 17:33:33.526656 29591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:33.532428 29591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:33.536867 29591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:33.537130 29591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:33.555615 29591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38439
I20260923 17:33:33.555660 29657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38439 every 8 connection(s)
I20260923 17:33:33.557022 29591 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:33.560778 29658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:33.561033 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 29591
I20260923 17:33:33.561203 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:33.561621 8311 external_mini_cluster.cc:1428] Setting key 64828e6ddb83ebf1ef39a182ad8a8cf1
I20260923 17:33:33.570206 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:33.563397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:43533 (local address 127.0.0.1:38439)
0923 17:33:33.564207 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:33.564220 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:33.564262 (+ 42us) server_negotiation.cc:407] Connection header received
0923 17:33:33.566696 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:33.566722 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:33.567111 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:33.567476 (+ 365us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:33.568573 (+ 1097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:33.568579 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:33.568601 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:33.568617 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:33.568715 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:33.568737 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:33.568905 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:33.569035 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 17:33:33.569234 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":351,"threads_started":1}
I20260923 17:33:33.575011 29658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f: Bootstrap starting.
I20260923 17:33:33.575117 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:33.578711 29658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:33.580076 29658 log.cc:824] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:33.582273 29661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:33.579620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:34875 (local address 127.0.0.1:38439)
0923 17:33:33.579847 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:33.579853 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:33.580157 (+ 304us) server_negotiation.cc:407] Connection header received
0923 17:33:33.580385 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:33.580390 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:33.580444 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:33.580526 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:33.581631 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:33.581636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:33.581639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:33.581655 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:33.581709 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:33.581726 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:33.581827 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:33.581965 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 17:33:33.582019 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260923 17:33:33.583505 29658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f: No bootstrap required, opened a new log
I20260923 17:33:33.591348 29658 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9fbfa6c6584e49a1fe7e6324add29f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38439 } }
I20260923 17:33:33.591812 29658 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:33.591924 29658 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d9fbfa6c6584e49a1fe7e6324add29f, State: Initialized, Role: FOLLOWER
I20260923 17:33:33.592613 29658 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9fbfa6c6584e49a1fe7e6324add29f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38439 } }
I20260923 17:33:33.592886 29658 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:33.593035 29658 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:33.593202 29658 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:33.595736 29658 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9fbfa6c6584e49a1fe7e6324add29f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38439 } }
I20260923 17:33:33.596312 29658 leader_election.cc:302] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d9fbfa6c6584e49a1fe7e6324add29f; no voters:
I20260923 17:33:33.596971 29658 leader_election.cc:288] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:33.597103 29668 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:33.597759 29668 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [term 1 LEADER]: Becoming Leader. State: Replica: 2d9fbfa6c6584e49a1fe7e6324add29f, State: Running, Role: LEADER
I20260923 17:33:33.598362 29668 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9fbfa6c6584e49a1fe7e6324add29f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38439 } }
I20260923 17:33:33.599242 29658 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:33.600917 29669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d9fbfa6c6584e49a1fe7e6324add29f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9fbfa6c6584e49a1fe7e6324add29f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38439 } } }
I20260923 17:33:33.601438 29669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:33.601225 29670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d9fbfa6c6584e49a1fe7e6324add29f. Latest consensus state: current_term: 1 leader_uuid: "2d9fbfa6c6584e49a1fe7e6324add29f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d9fbfa6c6584e49a1fe7e6324add29f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38439 } } }
I20260923 17:33:33.601670 29670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:33.604384 29673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:33.609153 29673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:33.618860 29673 catalog_manager.cc:1347] Generated new cluster ID: 2465abed0efa4e2494f3b7165f9b7d5c
I20260923 17:33:33.618988 29673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:33.638325 29673 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:33.638768 29673 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:33.644938 29673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d9fbfa6c6584e49a1fe7e6324add29f: Generated new TSK 0
I20260923 17:33:33.645843 29673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:33.658046 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29591
W20260923 17:33:33.659162 29653 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:43533
W20260923 17:33:33.659854 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29591: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:33.659974 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 29591
2026-09-23T17:33:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1818 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:33 dist-test-slave-p7jt krb5kdc[29689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:33 dist-test-slave-p7jt krb5kdc[29689](info): set up 2 sockets
Sep 23 17:33:33 dist-test-slave-p7jt krb5kdc[29689](info): commencing operation
krb5kdc: starting...
W20260923 17:33:34.686828 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.993s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:34 dist-test-slave-p7jt krb5kdc[29689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:34.945667 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33651
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:35.273602 29702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:35.273928 29702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:35.274065 29702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:35.281347 29702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:35.281481 29702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:35.281561 29702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:35.281652 29702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:35.281711 29702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:35.291690 29702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33651
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:35.293016 29702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:35.294323 29702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:35.305887 29710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:35.305917 29707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:35.305938 29708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:35.306931 29702 server_base.cc:1034] running on GCE node
I20260923 17:33:35.307713 29702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:35.309144 29702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:35.310362 29702 hybrid_clock.cc:648] HybridClock initialized: now 1790184815310316 us; error 38 us; skew 500 ppm
I20260923 17:33:35.310731 29702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:35 dist-test-slave-p7jt krb5kdc[29689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:35.318506 29702 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:35.320139 29702 webserver.cc:460] Webserver started at http://127.0.0.1:35115/ using document root <none> and password file <none>
I20260923 17:33:35.320847 29702 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:35.320962 29702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:35.321266 29702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:35.324292 29702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "84d4c271ee5045a9b91c4c8a6660241c"
format_stamp: "Formatted at 2026-09-23 17:33:35 on dist-test-slave-p7jt"
server_key: "43f1deb13656ac0561b28e95e6c276e2"
server_key_iv: "a1b324cd972bdcc08c07f22cb8857285"
server_key_version: "encryptionkey@0"
I20260923 17:33:35.325029 29702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "84d4c271ee5045a9b91c4c8a6660241c"
format_stamp: "Formatted at 2026-09-23 17:33:35 on dist-test-slave-p7jt"
server_key: "43f1deb13656ac0561b28e95e6c276e2"
server_key_iv: "a1b324cd972bdcc08c07f22cb8857285"
server_key_version: "encryptionkey@0"
I20260923 17:33:35.330250 29702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:33:35.333909 29717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:35.335335 29702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:35.335500 29702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "84d4c271ee5045a9b91c4c8a6660241c"
format_stamp: "Formatted at 2026-09-23 17:33:35 on dist-test-slave-p7jt"
server_key: "43f1deb13656ac0561b28e95e6c276e2"
server_key_iv: "a1b324cd972bdcc08c07f22cb8857285"
server_key_version: "encryptionkey@0"
I20260923 17:33:35.335680 29702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:35.356566 29702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:35.360898 29702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:35.361158 29702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:35.378257 29702 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33651
I20260923 17:33:35.378279 29768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33651 every 8 connection(s)
I20260923 17:33:35.379617 29702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:35.380175 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 29702
I20260923 17:33:35.380371 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:35.380937 8311 external_mini_cluster.cc:1428] Setting key 69dbf49b1c7c862f4b98a4bfcce85cc8
I20260923 17:33:35.383877 29769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 23 17:33:35 dist-test-slave-p7jt krb5kdc[29689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 17:33:35.394610 29769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c: Bootstrap starting.
I20260923 17:33:35.397877 29769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:35.399055 29769 log.cc:824] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:35.402334 29769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c: No bootstrap required, opened a new log
I20260923 17:33:35.403251 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:35.382738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:43973 (local address 127.0.0.1:33651)
0923 17:33:35.383454 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:35.383466 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:35.383507 (+ 41us) server_negotiation.cc:407] Connection header received
0923 17:33:35.385195 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:35.385217 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:35.385606 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:35.385946 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:35.387418 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:35.388787 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:35.390083 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:35.390826 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:35.390995 (+ 169us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:35.395332 (+ 4337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:35.395341 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:35.395359 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:35.395407 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:35.398739 (+ 3332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:35.399816 (+ 1077us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:35.399826 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:35.399839 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:35.400074 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:35.400524 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:35.400528 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:35.400530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:35.401059 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:35.401306 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:35.401821 (+ 515us) server_negotiation.cc:299] Negotiation successful
0923 17:33:35.402085 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":297,"threads_started":1}
I20260923 17:33:35.407361 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:35.412113 29769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c271ee5045a9b91c4c8a6660241c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } }
I20260923 17:33:35.412561 29769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:35.412721 29769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84d4c271ee5045a9b91c4c8a6660241c, State: Initialized, Role: FOLLOWER
I20260923 17:33:35.413381 29769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c271ee5045a9b91c4c8a6660241c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } }
I20260923 17:33:35.413671 29769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:35.413803 29769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:35.413980 29769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:35.416836 29769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c271ee5045a9b91c4c8a6660241c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } }
I20260923 17:33:35.417411 29769 leader_election.cc:302] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84d4c271ee5045a9b91c4c8a6660241c; no voters:
I20260923 17:33:35.418075 29769 leader_election.cc:288] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:35.418174 29779 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:35.418771 29779 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [term 1 LEADER]: Becoming Leader. State: Replica: 84d4c271ee5045a9b91c4c8a6660241c, State: Running, Role: LEADER
I20260923 17:33:35.419399 29779 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c271ee5045a9b91c4c8a6660241c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } }
I20260923 17:33:35.421419 29779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84d4c271ee5045a9b91c4c8a6660241c. Latest consensus state: current_term: 1 leader_uuid: "84d4c271ee5045a9b91c4c8a6660241c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c271ee5045a9b91c4c8a6660241c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } } }
I20260923 17:33:35.423012 29779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:35.422646 29781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84d4c271ee5045a9b91c4c8a6660241c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84d4c271ee5045a9b91c4c8a6660241c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33651 } } }
I20260923 17:33:35.423429 29781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:35.426126 29769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:35.426324 29783 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:35.427814 29772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:35.412449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:58759 (local address 127.0.0.1:33651)
0923 17:33:35.412642 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:35.412649 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:35.413049 (+ 400us) server_negotiation.cc:407] Connection header received
0923 17:33:35.413219 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:35.413223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:35.413285 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:35.413363 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:35.415258 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:35.416035 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:35.417681 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:35.418193 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:35.418329 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:35.420877 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:35.420882 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:35.420885 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:35.420915 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:35.424552 (+ 3637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:35.425442 (+ 890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:35.425448 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:35.425452 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:35.425625 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:35.426121 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:35.426126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:35.426129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:35.426547 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:35.426823 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:35.427273 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 17:33:35.427451 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260923 17:33:35.431608 29783 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:35.451819 29783 catalog_manager.cc:1347] Generated new cluster ID: 5970f40efc6e459da79556b4ce6e21e9
I20260923 17:33:35.451951 29783 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:35.476364 29783 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:35.477846 29783 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:35.489218 29783 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84d4c271ee5045a9b91c4c8a6660241c: Generated new TSK 0
I20260923 17:33:35.490155 29783 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:35.510739 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29702
I20260923 17:33:35.512004 29764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:35.607591 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 29702
W20260923 17:33:35.622354 29687 connection.cc:535] client connection to 127.0.0.1:33651 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33651): Connection reset by peer (error 104)
2026-09-23T17:33:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1947 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:35 dist-test-slave-p7jt krb5kdc[29801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:35 dist-test-slave-p7jt krb5kdc[29801](info): set up 2 sockets
Sep 23 17:33:35 dist-test-slave-p7jt krb5kdc[29801](info): commencing operation
krb5kdc: starting...
W20260923 17:33:36.634744 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.994s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:36 dist-test-slave-p7jt krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184816, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:36.890321 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42965
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:37.214947 29814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:37.215251 29814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:37.215390 29814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:37.222805 29814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:37.222944 29814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:37.223021 29814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:37.223109 29814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:37.223162 29814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:37.233494 29814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42965
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:37.234874 29814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:37.236194 29814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:37.247818 29822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:37.247768 29819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:37.247870 29820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:37.248396 29814 server_base.cc:1034] running on GCE node
I20260923 17:33:37.249181 29814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:37.250712 29814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:37.251914 29814 hybrid_clock.cc:648] HybridClock initialized: now 1790184817251866 us; error 47 us; skew 500 ppm
I20260923 17:33:37.252244 29814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:37 dist-test-slave-p7jt krb5kdc[29801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:37.255337 29814 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:37.256753 29814 webserver.cc:460] Webserver started at http://127.0.0.1:44021/ using document root <none> and password file <none>
I20260923 17:33:37.257412 29814 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:37.257526 29814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:37.257818 29814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:37.260872 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "b4e1ae86ee514331a26ce627c9a651bf"
format_stamp: "Formatted at 2026-09-23 17:33:37 on dist-test-slave-p7jt"
server_key: "1d6d889bd2ffdb26a53c1240c183c0ca"
server_key_iv: "14d5775675137419ddefc01698073a84"
server_key_version: "encryptionkey@0"
I20260923 17:33:37.261655 29814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "b4e1ae86ee514331a26ce627c9a651bf"
format_stamp: "Formatted at 2026-09-23 17:33:37 on dist-test-slave-p7jt"
server_key: "1d6d889bd2ffdb26a53c1240c183c0ca"
server_key_iv: "14d5775675137419ddefc01698073a84"
server_key_version: "encryptionkey@0"
I20260923 17:33:37.267053 29814 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:33:37.270779 29829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:37.272213 29814 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:37.272406 29814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "b4e1ae86ee514331a26ce627c9a651bf"
format_stamp: "Formatted at 2026-09-23 17:33:37 on dist-test-slave-p7jt"
server_key: "1d6d889bd2ffdb26a53c1240c183c0ca"
server_key_iv: "14d5775675137419ddefc01698073a84"
server_key_version: "encryptionkey@0"
I20260923 17:33:37.272604 29814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:37.289476 29814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:37.293839 29814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:37.294112 29814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:37.311822 29814 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42965
I20260923 17:33:37.311846 29880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42965 every 8 connection(s)
I20260923 17:33:37.313225 29814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:37.314857 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 29814
I20260923 17:33:37.315033 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:37.315563 8311 external_mini_cluster.cc:1428] Setting key 3747a2b1f8d5f10c8f16386aeba9eae0
I20260923 17:33:37.317809 29881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:37.329057 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:37.317316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:38249 (local address 127.0.0.1:42965)
0923 17:33:37.318158 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:37.318170 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:37.318209 (+ 39us) server_negotiation.cc:407] Connection header received
0923 17:33:37.319812 (+ 1603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:37.319833 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:37.320251 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:37.320587 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:37.321175 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:37.322593 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:37.324447 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:37.325126 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:37.325269 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:37.326610 (+ 1341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:37.326619 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:37.326635 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:37.327212 (+ 577us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:37.327328 (+ 116us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:37.327357 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:37.327492 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:37.327821 (+ 329us) server_negotiation.cc:299] Negotiation successful
0923 17:33:37.328042 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":636,"thread_start_us":419,"threads_started":1}
I20260923 17:33:37.330487 29881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf: Bootstrap starting.
I20260923 17:33:37.333288 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:37.333911 29881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:37.334966 29881 log.cc:824] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:37.337960 29881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf: No bootstrap required, opened a new log
I20260923 17:33:37.344173 29884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:37.337610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:56949 (local address 127.0.0.1:42965)
0923 17:33:37.337801 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:37.337808 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:37.338100 (+ 292us) server_negotiation.cc:407] Connection header received
0923 17:33:37.338323 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:37.338328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:37.338395 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:37.338516 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:37.339101 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:37.339870 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:37.341692 (+ 1822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:37.342159 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:37.342269 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:37.343443 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:37.343448 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:37.343451 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:37.343468 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:37.343528 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:37.343544 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:37.343627 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:37.343813 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 17:33:37.343864 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:33:37.347024 29881 raft_consensus.cc:348] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1ae86ee514331a26ce627c9a651bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } }
I20260923 17:33:37.347429 29881 raft_consensus.cc:374] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:37.347519 29881 raft_consensus.cc:729] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b4e1ae86ee514331a26ce627c9a651bf, State: Initialized, Role: FOLLOWER
I20260923 17:33:37.348079 29881 consensus_queue.cc:260] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1ae86ee514331a26ce627c9a651bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } }
I20260923 17:33:37.348306 29881 raft_consensus.cc:388] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:37.348430 29881 raft_consensus.cc:482] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:37.348587 29881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:37.350840 29881 raft_consensus.cc:504] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1ae86ee514331a26ce627c9a651bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } }
I20260923 17:33:37.351372 29881 leader_election.cc:302] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b4e1ae86ee514331a26ce627c9a651bf; no voters:
I20260923 17:33:37.351994 29881 leader_election.cc:288] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:37.352105 29891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:37.352763 29891 raft_consensus.cc:686] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [term 1 LEADER]: Becoming Leader. State: Replica: b4e1ae86ee514331a26ce627c9a651bf, State: Running, Role: LEADER
I20260923 17:33:37.353305 29891 consensus_queue.cc:237] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1ae86ee514331a26ce627c9a651bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } }
I20260923 17:33:37.354032 29881 sys_catalog.cc:564] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:37.356102 29892 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b4e1ae86ee514331a26ce627c9a651bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1ae86ee514331a26ce627c9a651bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } } }
I20260923 17:33:37.356554 29892 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:37.356829 29893 sys_catalog.cc:455] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b4e1ae86ee514331a26ce627c9a651bf. Latest consensus state: current_term: 1 leader_uuid: "b4e1ae86ee514331a26ce627c9a651bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b4e1ae86ee514331a26ce627c9a651bf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42965 } } }
I20260923 17:33:37.357152 29893 sys_catalog.cc:458] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:37.359439 29900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:37.364058 29900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:37.372244 29900 catalog_manager.cc:1347] Generated new cluster ID: 139fcbb66d574be58c765f5630eb2386
I20260923 17:33:37.372381 29900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:37.384136 29900 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:37.385478 29900 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:37.398731 29900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b4e1ae86ee514331a26ce627c9a651bf: Generated new TSK 0
I20260923 17:33:37.399623 29900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:37.410449 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29814
W20260923 17:33:37.411619 29876 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:38249
W20260923 17:33:37.412325 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29814: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:37.412489 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 29814
W20260923 17:33:37.426762 29799 connection.cc:535] client connection to 127.0.0.1:42965 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42965): Connection reset by peer (error 104)
2026-09-23T17:33:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1808 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:37 dist-test-slave-p7jt krb5kdc[29912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:37 dist-test-slave-p7jt krb5kdc[29912](info): set up 2 sockets
Sep 23 17:33:37 dist-test-slave-p7jt krb5kdc[29912](info): commencing operation
krb5kdc: starting...
W20260923 17:33:38.438685 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:38 dist-test-slave-p7jt krb5kdc[29912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 17:33:38.693841 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37367
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:39.022099 29925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:39.022392 29925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:39.022503 29925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:39.029881 29925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:39.030005 29925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:39.030051 29925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:39.030110 29925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:39.030153 29925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:39.040692 29925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:39.042023 29925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:39.043443 29925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:39.055162 29930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:39.055094 29931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:39.055115 29933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:39.056083 29925 server_base.cc:1034] running on GCE node
I20260923 17:33:39.056921 29925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:39.058429 29925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:39.059687 29925 hybrid_clock.cc:648] HybridClock initialized: now 1790184819059638 us; error 42 us; skew 500 ppm
I20260923 17:33:39.060055 29925 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:39 dist-test-slave-p7jt krb5kdc[29912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:39.063252 29925 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 17:33:39.064831 29925 webserver.cc:460] Webserver started at http://127.0.0.1:40381/ using document root <none> and password file <none>
I20260923 17:33:39.065507 29925 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:39.065625 29925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:39.065923 29925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:39.068972 29925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "46d649483fbc4e4ba049c5677b1d5f0c"
format_stamp: "Formatted at 2026-09-23 17:33:39 on dist-test-slave-p7jt"
server_key: "4065fd56f1e447fd4f1951933631425b"
server_key_iv: "d8143f0c2230cae35927b1bd68d58056"
server_key_version: "encryptionkey@0"
I20260923 17:33:39.069756 29925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "46d649483fbc4e4ba049c5677b1d5f0c"
format_stamp: "Formatted at 2026-09-23 17:33:39 on dist-test-slave-p7jt"
server_key: "4065fd56f1e447fd4f1951933631425b"
server_key_iv: "d8143f0c2230cae35927b1bd68d58056"
server_key_version: "encryptionkey@0"
I20260923 17:33:39.075542 29925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260923 17:33:39.079245 29940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:39.080689 29925 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:39.080868 29925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "46d649483fbc4e4ba049c5677b1d5f0c"
format_stamp: "Formatted at 2026-09-23 17:33:39 on dist-test-slave-p7jt"
server_key: "4065fd56f1e447fd4f1951933631425b"
server_key_iv: "d8143f0c2230cae35927b1bd68d58056"
server_key_version: "encryptionkey@0"
I20260923 17:33:39.081067 29925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:39.087083 29925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:39.091346 29925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:39.091620 29925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:39.109422 29925 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37367
I20260923 17:33:39.109455 29991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37367 every 8 connection(s)
I20260923 17:33:39.110932 29925 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:39.114650 29992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:39.117410 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 29925
I20260923 17:33:39.117594 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:39.118116 8311 external_mini_cluster.cc:1428] Setting key 6a4fd77cdbce6dd765337bb91c1b6871
I20260923 17:33:39.126140 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:39.120029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:38513 (local address 127.0.0.1:37367)
0923 17:33:39.121036 (+ 1007us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:39.121048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:39.121088 (+ 40us) server_negotiation.cc:407] Connection header received
0923 17:33:39.122828 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:39.122851 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:39.123197 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:39.123526 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:39.124581 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:39.124591 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:39.124604 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:39.124620 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:39.124720 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:39.124745 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:39.124914 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:39.125016 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 17:33:39.125207 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":368,"threads_started":1}
I20260923 17:33:39.127064 29992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c: Bootstrap starting.
I20260923 17:33:39.129729 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:39.130378 29992 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:39.131628 29992 log.cc:824] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:39.135049 29992 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c: No bootstrap required, opened a new log
I20260923 17:33:39.137452 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:39.134896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.185:39311 (local address 127.0.0.1:37367)
0923 17:33:39.135086 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:39.135093 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:39.135538 (+ 445us) server_negotiation.cc:407] Connection header received
0923 17:33:39.135721 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:39.135727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:39.135789 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:39.135911 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:39.136831 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:39.136838 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:39.136842 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:39.136865 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:39.136921 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:39.136941 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:39.137058 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:39.137101 (+ 43us) server_negotiation.cc:299] Negotiation successful
0923 17:33:39.137171 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:33:39.143170 29992 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d649483fbc4e4ba049c5677b1d5f0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260923 17:33:39.143536 29992 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:39.143607 29992 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d649483fbc4e4ba049c5677b1d5f0c, State: Initialized, Role: FOLLOWER
I20260923 17:33:39.144114 29992 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d649483fbc4e4ba049c5677b1d5f0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260923 17:33:39.144342 29992 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:39.144479 29992 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:39.144670 29992 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:39.147105 29992 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d649483fbc4e4ba049c5677b1d5f0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260923 17:33:39.147614 29992 leader_election.cc:302] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46d649483fbc4e4ba049c5677b1d5f0c; no voters:
I20260923 17:33:39.148335 29992 leader_election.cc:288] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:39.148478 30002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:39.149196 30002 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [term 1 LEADER]: Becoming Leader. State: Replica: 46d649483fbc4e4ba049c5677b1d5f0c, State: Running, Role: LEADER
I20260923 17:33:39.149725 30002 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d649483fbc4e4ba049c5677b1d5f0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } }
I20260923 17:33:39.150501 29992 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:39.152592 30003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46d649483fbc4e4ba049c5677b1d5f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d649483fbc4e4ba049c5677b1d5f0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } } }
I20260923 17:33:39.154043 30003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:39.153215 30004 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46d649483fbc4e4ba049c5677b1d5f0c. Latest consensus state: current_term: 1 leader_uuid: "46d649483fbc4e4ba049c5677b1d5f0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d649483fbc4e4ba049c5677b1d5f0c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37367 } } }
I20260923 17:33:39.155164 30004 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:39.155273 30012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:39.160418 30012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:39.170537 30012 catalog_manager.cc:1347] Generated new cluster ID: 1234347e01dd403fbbb99a2b702eeadd
I20260923 17:33:39.170667 30012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:39.188350 30012 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:39.188774 30012 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:39.199990 30012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46d649483fbc4e4ba049c5677b1d5f0c: Generated new TSK 0
I20260923 17:33:39.200824 30012 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:39.205209 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29925
W20260923 17:33:39.206404 29987 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.185:38513
W20260923 17:33:39.207325 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 29925: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:39.207480 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 29925
2026-09-23T17:33:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1790 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:39 dist-test-slave-p7jt krb5kdc[30023](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:39 dist-test-slave-p7jt krb5kdc[30023](info): set up 2 sockets
Sep 23 17:33:39 dist-test-slave-p7jt krb5kdc[30023](info): commencing operation
krb5kdc: starting...
W20260923 17:33:40.226747 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.986s user 0.005s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:40 dist-test-slave-p7jt krb5kdc[30023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:40.483621 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44809
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:44809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:40.816215 30036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:40.816536 30036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:40.816679 30036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:40.824007 30036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:40.824132 30036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:40.824211 30036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:40.824297 30036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:40.824357 30036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:40.834386 30036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:44809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:40.835712 30036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:40.836997 30036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:40.848398 30042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:40.848467 30041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:40.848448 30044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:40.849488 30036 server_base.cc:1034] running on GCE node
I20260923 17:33:40.850310 30036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:40.851910 30036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:40.853150 30036 hybrid_clock.cc:648] HybridClock initialized: now 1790184820853094 us; error 50 us; skew 500 ppm
I20260923 17:33:40.853525 30036 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:40 dist-test-slave-p7jt krb5kdc[30023](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184820, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:40.861938 30036 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:40.863597 30036 webserver.cc:460] Webserver started at http://127.8.29.254:41647/ using document root <none> and password file <none>
I20260923 17:33:40.864293 30036 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:40.864414 30036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:40.864701 30036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:40.867848 30036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "731d98c84ee54c558111a6d9bae4a92e"
format_stamp: "Formatted at 2026-09-23 17:33:40 on dist-test-slave-p7jt"
server_key: "514910bcf63c2c07e7c4b10326550c04"
server_key_iv: "09c48cb6f05f3883328f4f12f588df6b"
server_key_version: "encryptionkey@0"
I20260923 17:33:40.868599 30036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "731d98c84ee54c558111a6d9bae4a92e"
format_stamp: "Formatted at 2026-09-23 17:33:40 on dist-test-slave-p7jt"
server_key: "514910bcf63c2c07e7c4b10326550c04"
server_key_iv: "09c48cb6f05f3883328f4f12f588df6b"
server_key_version: "encryptionkey@0"
I20260923 17:33:40.873943 30036 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260923 17:33:40.877553 30051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:40.878957 30036 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:40.879129 30036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "731d98c84ee54c558111a6d9bae4a92e"
format_stamp: "Formatted at 2026-09-23 17:33:40 on dist-test-slave-p7jt"
server_key: "514910bcf63c2c07e7c4b10326550c04"
server_key_iv: "09c48cb6f05f3883328f4f12f588df6b"
server_key_version: "encryptionkey@0"
I20260923 17:33:40.879317 30036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:40.896188 30036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:40.900348 30036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:40.900615 30036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:40.917935 30036 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:44809
I20260923 17:33:40.917958 30102 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:44809 every 8 connection(s)
I20260923 17:33:40.919342 30036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:40.923058 30103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:40.928544 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 30036
I20260923 17:33:40.928722 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:40.929132 8311 external_mini_cluster.cc:1428] Setting key 7b633a96dc16062dcdee9b290c7f262e
I20260923 17:33:40.934983 30103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e: Bootstrap starting.
I20260923 17:33:40.938853 30103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:40.939929 30103 log.cc:824] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:40.943717 30103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e: No bootstrap required, opened a new log
Sep 23 17:33:40 dist-test-slave-p7jt krb5kdc[30023](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:40.952860 30103 raft_consensus.cc:348] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731d98c84ee54c558111a6d9bae4a92e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44809 } }
I20260923 17:33:40.953320 30103 raft_consensus.cc:374] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:40.953429 30103 raft_consensus.cc:729] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 731d98c84ee54c558111a6d9bae4a92e, State: Initialized, Role: FOLLOWER
I20260923 17:33:40.954005 30103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731d98c84ee54c558111a6d9bae4a92e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44809 } }
I20260923 17:33:40.954247 30103 raft_consensus.cc:388] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:40.954388 30103 raft_consensus.cc:482] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:40.954447 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:40.931079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53859 (local address 127.8.29.254:44809)
0923 17:33:40.931978 (+ 899us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:40.931994 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:40.932053 (+ 59us) server_negotiation.cc:407] Connection header received
0923 17:33:40.934373 (+ 2320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:40.934402 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:40.934981 (+ 579us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:40.935428 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:40.937184 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:40.938609 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:40.940009 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:40.941038 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:40.941256 (+ 218us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:40.946419 (+ 5163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:40.946431 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:40.946449 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:40.946529 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:40.949677 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:40.950772 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:40.950783 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:40.950794 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:40.951002 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:40.951541 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:40.951545 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:40.951547 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:40.952155 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:40.952408 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:40.952886 (+ 478us) server_negotiation.cc:299] Negotiation successful
0923 17:33:40.953197 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":354,"threads_started":1}
I20260923 17:33:40.954602 30103 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:40.956972 30103 raft_consensus.cc:504] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731d98c84ee54c558111a6d9bae4a92e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44809 } }
I20260923 17:33:40.957568 30103 leader_election.cc:302] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 731d98c84ee54c558111a6d9bae4a92e; no voters:
I20260923 17:33:40.958227 30103 leader_election.cc:288] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:40.958330 30108 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:40.958746 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:40.960386 30103 sys_catalog.cc:564] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:40.963338 30108 raft_consensus.cc:686] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [term 1 LEADER]: Becoming Leader. State: Replica: 731d98c84ee54c558111a6d9bae4a92e, State: Running, Role: LEADER
I20260923 17:33:40.963992 30108 consensus_queue.cc:237] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731d98c84ee54c558111a6d9bae4a92e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44809 } }
I20260923 17:33:40.966604 30109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "731d98c84ee54c558111a6d9bae4a92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731d98c84ee54c558111a6d9bae4a92e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44809 } } }
I20260923 17:33:40.967046 30109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:40.967536 30110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 731d98c84ee54c558111a6d9bae4a92e. Latest consensus state: current_term: 1 leader_uuid: "731d98c84ee54c558111a6d9bae4a92e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "731d98c84ee54c558111a6d9bae4a92e" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44809 } } }
I20260923 17:33:40.967818 30110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:40.970196 30117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:40.977028 30117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:41.001334 30117 catalog_manager.cc:1347] Generated new cluster ID: e408467383ab49d8adcbc8b08144eb48
I20260923 17:33:41.001474 30117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:41.001367 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:40.972869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58035 (local address 127.8.29.254:44809)
0923 17:33:40.973077 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:40.973085 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:40.983430 (+ 10345us) server_negotiation.cc:407] Connection header received
0923 17:33:40.983739 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:40.983746 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:40.983850 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:40.983925 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:40.986052 (+ 2127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:40.987185 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:40.989029 (+ 1844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:40.989868 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:40.990024 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:40.993427 (+ 3403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:40.993435 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:40.993440 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:40.993489 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:40.997315 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:40.998545 (+ 1230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:40.998552 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:40.998556 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:40.998723 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:40.999335 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:40.999340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:40.999343 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:40.999825 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:41.000084 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:41.000795 (+ 711us) server_negotiation.cc:299] Negotiation successful
0923 17:33:41.000987 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:33:41.035506 30117 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:41.036841 30117 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:41.054023 30117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 731d98c84ee54c558111a6d9bae4a92e: Generated new TSK 0
I20260923 17:33:41.054946 30117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:41.062624 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30036
I20260923 17:33:41.063874 30098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:41.157440 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 30036
W20260923 17:33:41.171813 30021 connection.cc:535] client connection to 127.8.29.254:44809 recv error: Network error: failed to read from TLS socket (remote: 127.8.29.254:44809): Connection reset by peer (error 104)
2026-09-23T17:33:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1949 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:41 dist-test-slave-p7jt krb5kdc[30136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:41 dist-test-slave-p7jt krb5kdc[30136](info): set up 2 sockets
Sep 23 17:33:41 dist-test-slave-p7jt krb5kdc[30136](info): commencing operation
krb5kdc: starting...
W20260923 17:33:42.174747 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.985s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:42 dist-test-slave-p7jt krb5kdc[30136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184822, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:42.423959 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40241
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:40241
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:42.753633 30149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:42.753957 30149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:42.754069 30149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:42.762558 30149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:42.762676 30149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:42.762748 30149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:42.762826 30149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:42.762883 30149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:42.773312 30149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:40241
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:40241
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:42.774679 30149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:42.775954 30149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:42.787695 30157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:42.787729 30154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:42.787765 30155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:42.788761 30149 server_base.cc:1034] running on GCE node
I20260923 17:33:42.789637 30149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:42.791296 30149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:42.792542 30149 hybrid_clock.cc:648] HybridClock initialized: now 1790184822792494 us; error 47 us; skew 500 ppm
I20260923 17:33:42.792905 30149 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:42 dist-test-slave-p7jt krb5kdc[30136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184822, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:42.795969 30149 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:42.797506 30149 webserver.cc:460] Webserver started at http://127.8.29.254:44919/ using document root <none> and password file <none>
I20260923 17:33:42.798177 30149 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:42.798302 30149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:42.798631 30149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:42.801702 30149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "5e13528743c748e3b14e7470c6a82492"
format_stamp: "Formatted at 2026-09-23 17:33:42 on dist-test-slave-p7jt"
server_key: "aac29799f84a29660a13d8edcaf1016d"
server_key_iv: "dd0de96f1ee9a778e477ee105ee1284e"
server_key_version: "encryptionkey@0"
I20260923 17:33:42.802556 30149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "5e13528743c748e3b14e7470c6a82492"
format_stamp: "Formatted at 2026-09-23 17:33:42 on dist-test-slave-p7jt"
server_key: "aac29799f84a29660a13d8edcaf1016d"
server_key_iv: "dd0de96f1ee9a778e477ee105ee1284e"
server_key_version: "encryptionkey@0"
I20260923 17:33:42.807988 30149 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260923 17:33:42.811757 30164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:42.813182 30149 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:33:42.813373 30149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "5e13528743c748e3b14e7470c6a82492"
format_stamp: "Formatted at 2026-09-23 17:33:42 on dist-test-slave-p7jt"
server_key: "aac29799f84a29660a13d8edcaf1016d"
server_key_iv: "dd0de96f1ee9a778e477ee105ee1284e"
server_key_version: "encryptionkey@0"
I20260923 17:33:42.813570 30149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:42.838387 30149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:42.842621 30149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:42.842875 30149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:42.861485 30149 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:40241
I20260923 17:33:42.861528 30215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:40241 every 8 connection(s)
I20260923 17:33:42.862951 30149 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:42.866976 30216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:42.868182 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 30149
I20260923 17:33:42.868366 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:42.868845 8311 external_mini_cluster.cc:1428] Setting key 80e8bdb3d260034c2039f2c7e0db2b47
I20260923 17:33:42.881546 30216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492: Bootstrap starting.
I20260923 17:33:42.883481 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:42.870791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37289 (local address 127.8.29.254:40241)
0923 17:33:42.871631 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:42.871649 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:42.871704 (+ 55us) server_negotiation.cc:407] Connection header received
0923 17:33:42.873518 (+ 1814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:42.873553 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:42.874111 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:42.874608 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:42.875128 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:42.876940 (+ 1812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:42.878273 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:42.879378 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:42.879552 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:42.880705 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:42.880716 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:42.880735 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:42.881428 (+ 693us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:42.881540 (+ 112us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:42.881570 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:42.881719 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:42.881950 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 17:33:42.882131 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":344,"threads_started":1}
I20260923 17:33:42.885239 30216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:42.886350 30216 log.cc:824] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:42.887296 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:42.889541 30216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492: No bootstrap required, opened a new log
I20260923 17:33:42.898900 30219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:42.892335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39109 (local address 127.8.29.254:40241)
0923 17:33:42.892536 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:42.892545 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:42.892833 (+ 288us) server_negotiation.cc:407] Connection header received
0923 17:33:42.892979 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:42.892984 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:42.893063 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:42.893175 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:42.893617 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:42.894781 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:42.896127 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:42.896817 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:42.896953 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:42.898065 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:42.898071 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:42.898076 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:42.898097 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:42.898158 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:42.898180 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:42.898289 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:42.898532 (+ 243us) server_negotiation.cc:299] Negotiation successful
0923 17:33:42.898586 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260923 17:33:42.899219 30216 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e13528743c748e3b14e7470c6a82492" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40241 } }
I20260923 17:33:42.899631 30216 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:42.899731 30216 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e13528743c748e3b14e7470c6a82492, State: Initialized, Role: FOLLOWER
I20260923 17:33:42.900369 30216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e13528743c748e3b14e7470c6a82492" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40241 } }
I20260923 17:33:42.900604 30216 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:42.900740 30216 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:42.900887 30216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:42.903151 30216 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e13528743c748e3b14e7470c6a82492" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40241 } }
I20260923 17:33:42.903643 30216 leader_election.cc:302] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e13528743c748e3b14e7470c6a82492; no voters:
I20260923 17:33:42.904197 30216 leader_election.cc:288] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:42.904316 30226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:42.905853 30226 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [term 1 LEADER]: Becoming Leader. State: Replica: 5e13528743c748e3b14e7470c6a82492, State: Running, Role: LEADER
I20260923 17:33:42.906390 30216 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:42.906442 30226 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e13528743c748e3b14e7470c6a82492" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40241 } }
I20260923 17:33:42.909534 30227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e13528743c748e3b14e7470c6a82492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e13528743c748e3b14e7470c6a82492" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40241 } } }
I20260923 17:33:42.909514 30228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e13528743c748e3b14e7470c6a82492. Latest consensus state: current_term: 1 leader_uuid: "5e13528743c748e3b14e7470c6a82492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e13528743c748e3b14e7470c6a82492" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 40241 } } }
I20260923 17:33:42.910010 30227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:42.910012 30228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:42.916975 30235 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:42.921597 30235 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:42.931283 30235 catalog_manager.cc:1347] Generated new cluster ID: a572fe8e944e4ba58162e433550d3eee
I20260923 17:33:42.931417 30235 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:42.955905 30235 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:42.957218 30235 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:42.967940 30235 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e13528743c748e3b14e7470c6a82492: Generated new TSK 0
I20260923 17:33:42.969184 30235 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:42.986477 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30149
W20260923 17:33:42.987815 30211 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:37289
W20260923 17:33:42.988583 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30149: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:42.988742 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 30149
W20260923 17:33:43.002954 30134 connection.cc:535] client connection to 127.8.29.254:40241 recv error: Network error: failed to read from TLS socket (remote: 127.8.29.254:40241): Connection reset by peer (error 104)
2026-09-23T17:33:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1831 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:43 dist-test-slave-p7jt krb5kdc[30247](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:43 dist-test-slave-p7jt krb5kdc[30247](info): set up 2 sockets
Sep 23 17:33:43 dist-test-slave-p7jt krb5kdc[30247](info): commencing operation
krb5kdc: starting...
W20260923 17:33:44.022814 8311 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:44 dist-test-slave-p7jt krb5kdc[30247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:44.275596 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:46141
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39695
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:46141
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:44.603993 30260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:44.604305 30260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:44.604444 30260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:44.611645 30260 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:44.611771 30260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:44.611852 30260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:44.611929 30260 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:44.611989 30260 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:44.622061 30260 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39695
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:46141
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:46141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:44.623392 30260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:44.624751 30260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:44.636152 30265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:44.636185 30266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:44.636112 30268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:44.637322 30260 server_base.cc:1034] running on GCE node
I20260923 17:33:44.638104 30260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:44.639546 30260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:44.640779 30260 hybrid_clock.cc:648] HybridClock initialized: now 1790184824640718 us; error 48 us; skew 500 ppm
I20260923 17:33:44.641126 30260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:44 dist-test-slave-p7jt krb5kdc[30247](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:44.644230 30260 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:44.645694 30260 webserver.cc:460] Webserver started at http://127.8.29.254:33721/ using document root <none> and password file <none>
I20260923 17:33:44.646392 30260 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:44.646530 30260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:44.646827 30260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:44.649726 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "bc35c6bd371041fc8cf700b083ab4f23"
format_stamp: "Formatted at 2026-09-23 17:33:44 on dist-test-slave-p7jt"
server_key: "2876f101dff60999f794ddf546f7e45a"
server_key_iv: "5beb80125b21ee7c90672e80f6a22eba"
server_key_version: "encryptionkey@0"
I20260923 17:33:44.650480 30260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "bc35c6bd371041fc8cf700b083ab4f23"
format_stamp: "Formatted at 2026-09-23 17:33:44 on dist-test-slave-p7jt"
server_key: "2876f101dff60999f794ddf546f7e45a"
server_key_iv: "5beb80125b21ee7c90672e80f6a22eba"
server_key_version: "encryptionkey@0"
I20260923 17:33:44.655972 30260 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260923 17:33:44.659861 30275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:44.661298 30260 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:44.661490 30260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "bc35c6bd371041fc8cf700b083ab4f23"
format_stamp: "Formatted at 2026-09-23 17:33:44 on dist-test-slave-p7jt"
server_key: "2876f101dff60999f794ddf546f7e45a"
server_key_iv: "5beb80125b21ee7c90672e80f6a22eba"
server_key_version: "encryptionkey@0"
I20260923 17:33:44.661741 30260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:44.667973 30260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:44.672389 30260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:44.672686 30260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:44.689358 30260 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:46141
I20260923 17:33:44.689380 30326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:46141 every 8 connection(s)
I20260923 17:33:44.690943 30260 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:44.694979 30327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:44.699836 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 30260
I20260923 17:33:44.700007 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:44.700448 8311 external_mini_cluster.cc:1428] Setting key 025cdb2bf5dc23b3ddbef7df6cddce70
I20260923 17:33:44.707445 30327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23: Bootstrap starting.
I20260923 17:33:44.709559 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:44.702230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51445 (local address 127.8.29.254:46141)
0923 17:33:44.703070 (+ 840us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:44.703085 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:44.703143 (+ 58us) server_negotiation.cc:407] Connection header received
0923 17:33:44.705593 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:44.705624 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:44.706098 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:44.706613 (+ 515us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:44.707605 (+ 992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:44.707613 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:44.707630 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:44.707647 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:44.707716 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:44.707748 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:44.707965 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:44.708094 (+ 129us) server_negotiation.cc:299] Negotiation successful
0923 17:33:44.708310 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":356,"threads_started":1}
I20260923 17:33:44.711359 30327 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:44.712561 30327 log.cc:824] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:44.713421 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:44.715409 30327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23: No bootstrap required, opened a new log
I20260923 17:33:44.719913 30330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:44.717430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59835 (local address 127.8.29.254:46141)
0923 17:33:44.717606 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:44.717612 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:44.718014 (+ 402us) server_negotiation.cc:407] Connection header received
0923 17:33:44.718151 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:44.718156 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:44.718213 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:44.718330 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:44.719252 (+ 922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:44.719257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:44.719262 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:44.719280 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:44.719324 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:44.719342 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:44.719451 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:44.719554 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 17:33:44.719615 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:33:44.723433 30327 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35c6bd371041fc8cf700b083ab4f23" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46141 } }
I20260923 17:33:44.723773 30327 raft_consensus.cc:374] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:44.723846 30327 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc35c6bd371041fc8cf700b083ab4f23, State: Initialized, Role: FOLLOWER
I20260923 17:33:44.724326 30327 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35c6bd371041fc8cf700b083ab4f23" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46141 } }
I20260923 17:33:44.724534 30327 raft_consensus.cc:388] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:44.724659 30327 raft_consensus.cc:482] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:44.724820 30327 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:44.727111 30327 raft_consensus.cc:504] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35c6bd371041fc8cf700b083ab4f23" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46141 } }
I20260923 17:33:44.727553 30327 leader_election.cc:302] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bc35c6bd371041fc8cf700b083ab4f23; no voters:
I20260923 17:33:44.728168 30327 leader_election.cc:288] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:44.728317 30337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:44.729461 30337 raft_consensus.cc:686] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [term 1 LEADER]: Becoming Leader. State: Replica: bc35c6bd371041fc8cf700b083ab4f23, State: Running, Role: LEADER
I20260923 17:33:44.729988 30337 consensus_queue.cc:237] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35c6bd371041fc8cf700b083ab4f23" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46141 } }
I20260923 17:33:44.730217 30327 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:44.732532 30339 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bc35c6bd371041fc8cf700b083ab4f23. Latest consensus state: current_term: 1 leader_uuid: "bc35c6bd371041fc8cf700b083ab4f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35c6bd371041fc8cf700b083ab4f23" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46141 } } }
I20260923 17:33:44.732676 30338 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bc35c6bd371041fc8cf700b083ab4f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bc35c6bd371041fc8cf700b083ab4f23" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 46141 } } }
I20260923 17:33:44.732933 30339 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:44.732987 30338 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:44.734615 30346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:44.739624 30346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:44.748481 30346 catalog_manager.cc:1347] Generated new cluster ID: 88cc52ed8f23448585aae558b74439df
I20260923 17:33:44.748620 30346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:44.779150 30346 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:44.779683 30346 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:44.790781 30346 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bc35c6bd371041fc8cf700b083ab4f23: Generated new TSK 0
I20260923 17:33:44.791908 30346 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:44.797752 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30260
W20260923 17:33:44.798893 30322 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51445
W20260923 17:33:44.799573 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30260: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:44.799744 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 30260
2026-09-23T17:33:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1810 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:44 dist-test-slave-p7jt krb5kdc[30358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:44 dist-test-slave-p7jt krb5kdc[30358](info): set up 2 sockets
Sep 23 17:33:44 dist-test-slave-p7jt krb5kdc[30358](info): commencing operation
krb5kdc: starting...
W20260923 17:33:45.819032 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:45 dist-test-slave-p7jt krb5kdc[30358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:46.099074 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39725
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:40443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:39725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:46.427743 30371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:46.428043 30371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:46.428164 30371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:46.435515 30371 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:46.435631 30371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:46.435701 30371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:46.435784 30371 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:46.435846 30371 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:46.445912 30371 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:40443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:39725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:39725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:46.447227 30371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:46.448518 30371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:46.459924 30376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:46.459856 30379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:46.459905 30377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:46.460615 30371 server_base.cc:1034] running on GCE node
I20260923 17:33:46.461462 30371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:46.463017 30371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:46.464237 30371 hybrid_clock.cc:648] HybridClock initialized: now 1790184826464180 us; error 57 us; skew 500 ppm
I20260923 17:33:46.464570 30371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:46 dist-test-slave-p7jt krb5kdc[30358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:46.471396 30371 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:46.473022 30371 webserver.cc:460] Webserver started at http://127.8.29.254:45477/ using document root <none> and password file <none>
I20260923 17:33:46.473685 30371 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:46.473795 30371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:46.474067 30371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:46.477031 30371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "2d7c3f93415d4852abce563ab417687f"
format_stamp: "Formatted at 2026-09-23 17:33:46 on dist-test-slave-p7jt"
server_key: "a5e4004b8b3dd1e5f0ec5745b7b8fbd6"
server_key_iv: "dcbea23462d1f68a15f2fd5b3714c10e"
server_key_version: "encryptionkey@0"
I20260923 17:33:46.477746 30371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "2d7c3f93415d4852abce563ab417687f"
format_stamp: "Formatted at 2026-09-23 17:33:46 on dist-test-slave-p7jt"
server_key: "a5e4004b8b3dd1e5f0ec5745b7b8fbd6"
server_key_iv: "dcbea23462d1f68a15f2fd5b3714c10e"
server_key_version: "encryptionkey@0"
I20260923 17:33:46.483031 30371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:33:46.486780 30386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:46.488210 30371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:46.488374 30371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "2d7c3f93415d4852abce563ab417687f"
format_stamp: "Formatted at 2026-09-23 17:33:46 on dist-test-slave-p7jt"
server_key: "a5e4004b8b3dd1e5f0ec5745b7b8fbd6"
server_key_iv: "dcbea23462d1f68a15f2fd5b3714c10e"
server_key_version: "encryptionkey@0"
I20260923 17:33:46.488567 30371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:46.502736 30371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:46.507169 30371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:46.507431 30371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:46.527172 30371 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:39725
I20260923 17:33:46.527205 30437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:39725 every 8 connection(s)
I20260923 17:33:46.528565 30371 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:46.532573 30438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:46.533906 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 30371
I20260923 17:33:46.534072 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:46.534523 8311 external_mini_cluster.cc:1428] Setting key 8fce2a61a117fbcfdac67d6f9d92d1fc
I20260923 17:33:46.546900 30438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f: Bootstrap starting.
Sep 23 17:33:46 dist-test-slave-p7jt krb5kdc[30358](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184825, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:46.550841 30438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:46.552028 30438 log.cc:824] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:46.555375 30438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f: No bootstrap required, opened a new log
I20260923 17:33:46.558871 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:46.536477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58767 (local address 127.8.29.254:39725)
0923 17:33:46.537299 (+ 822us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:46.537311 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:46.537358 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:33:46.539025 (+ 1667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:46.539050 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:46.539496 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:46.539852 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:46.541580 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:46.542965 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:46.544778 (+ 1813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:46.545503 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:46.545655 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:46.550548 (+ 4893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:46.550559 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:46.550576 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:46.550625 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:46.554015 (+ 3390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:46.554830 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:46.554839 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:46.554847 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:46.555052 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:46.555447 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:46.555452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:46.555455 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:46.556185 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:46.556497 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:46.556935 (+ 438us) server_negotiation.cc:299] Negotiation successful
0923 17:33:46.557416 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":378,"threads_started":1}
I20260923 17:33:46.563062 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:46.564837 30438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c3f93415d4852abce563ab417687f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39725 } }
I20260923 17:33:46.565198 30438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:46.565335 30438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d7c3f93415d4852abce563ab417687f, State: Initialized, Role: FOLLOWER
I20260923 17:33:46.565934 30438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c3f93415d4852abce563ab417687f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39725 } }
I20260923 17:33:46.566198 30438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:46.566335 30438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:46.566527 30438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:46.569655 30438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c3f93415d4852abce563ab417687f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39725 } }
I20260923 17:33:46.570252 30438 leader_election.cc:302] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d7c3f93415d4852abce563ab417687f; no voters:
I20260923 17:33:46.570909 30438 leader_election.cc:288] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:46.571015 30448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:46.572211 30448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [term 1 LEADER]: Becoming Leader. State: Replica: 2d7c3f93415d4852abce563ab417687f, State: Running, Role: LEADER
I20260923 17:33:46.572916 30448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c3f93415d4852abce563ab417687f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39725 } }
I20260923 17:33:46.573390 30438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:46.575855 30449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d7c3f93415d4852abce563ab417687f. Latest consensus state: current_term: 1 leader_uuid: "2d7c3f93415d4852abce563ab417687f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c3f93415d4852abce563ab417687f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39725 } } }
I20260923 17:33:46.576257 30449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:46.578678 30450 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d7c3f93415d4852abce563ab417687f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d7c3f93415d4852abce563ab417687f" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 39725 } } }
I20260923 17:33:46.579069 30450 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:46.581212 30453 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:46.582863 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:46.567192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33593 (local address 127.8.29.254:39725)
0923 17:33:46.567388 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:46.567395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:46.567820 (+ 425us) server_negotiation.cc:407] Connection header received
0923 17:33:46.567979 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:46.567983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:46.568051 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:46.568133 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:46.569978 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:46.570867 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:46.572591 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:46.573137 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:46.573257 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:46.576198 (+ 2941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:46.576205 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:46.576210 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:46.576254 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:46.579361 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:46.580298 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:46.580308 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:46.580311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:46.580470 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:46.580975 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:46.580981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:46.580985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:46.581473 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:46.581749 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:46.582159 (+ 410us) server_negotiation.cc:299] Negotiation successful
0923 17:33:46.582427 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 17:33:46.586560 30453 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:46.596510 30453 catalog_manager.cc:1347] Generated new cluster ID: 2e9a120a775a4068854cdaa8394209d9
I20260923 17:33:46.596637 30453 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:46.612051 30453 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:46.613391 30453 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:46.621480 30453 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d7c3f93415d4852abce563ab417687f: Generated new TSK 0
I20260923 17:33:46.622383 30453 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:46.636584 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30371
I20260923 17:33:46.637684 30433 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:46.727578 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 30371
2026-09-23T17:33:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1928 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:46 dist-test-slave-p7jt krb5kdc[30470](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:46 dist-test-slave-p7jt krb5kdc[30470](info): set up 2 sockets
Sep 23 17:33:46 dist-test-slave-p7jt krb5kdc[30470](info): commencing operation
krb5kdc: starting...
W20260923 17:33:47.746729 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.987s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:47 dist-test-slave-p7jt krb5kdc[30470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:48.016227 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:36875
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:39351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:36875
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:48.341537 30483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:48.341876 30483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:48.342000 30483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:48.349371 30483 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:48.349504 30483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:48.349582 30483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:48.349668 30483 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:48.349735 30483 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:48.359781 30483 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:39351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:36875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:36875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:48.361210 30483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:48.362545 30483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:48.374049 30489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:48.374140 30491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:48.374166 30488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:48.374934 30483 server_base.cc:1034] running on GCE node
I20260923 17:33:48.375736 30483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:48.377141 30483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:48.378340 30483 hybrid_clock.cc:648] HybridClock initialized: now 1790184828378290 us; error 39 us; skew 500 ppm
I20260923 17:33:48.378702 30483 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:48 dist-test-slave-p7jt krb5kdc[30470](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184828, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:48.381721 30483 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:48.383188 30483 webserver.cc:460] Webserver started at http://127.8.29.254:46223/ using document root <none> and password file <none>
I20260923 17:33:48.383862 30483 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:48.383980 30483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:48.384271 30483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:48.387331 30483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "3aceb05ee3cf4240866f8135759cb4e2"
format_stamp: "Formatted at 2026-09-23 17:33:48 on dist-test-slave-p7jt"
server_key: "5e36d80d55a97be538bb1f5f3db59c78"
server_key_iv: "b8a45662f6d2b27d64e3865441475899"
server_key_version: "encryptionkey@0"
I20260923 17:33:48.388082 30483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "3aceb05ee3cf4240866f8135759cb4e2"
format_stamp: "Formatted at 2026-09-23 17:33:48 on dist-test-slave-p7jt"
server_key: "5e36d80d55a97be538bb1f5f3db59c78"
server_key_iv: "b8a45662f6d2b27d64e3865441475899"
server_key_version: "encryptionkey@0"
I20260923 17:33:48.393515 30483 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260923 17:33:48.397119 30498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:48.398505 30483 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260923 17:33:48.398671 30483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "3aceb05ee3cf4240866f8135759cb4e2"
format_stamp: "Formatted at 2026-09-23 17:33:48 on dist-test-slave-p7jt"
server_key: "5e36d80d55a97be538bb1f5f3db59c78"
server_key_iv: "b8a45662f6d2b27d64e3865441475899"
server_key_version: "encryptionkey@0"
I20260923 17:33:48.398851 30483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:48.408941 30483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:48.412796 30483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:48.413053 30483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:48.430624 30483 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:36875
I20260923 17:33:48.430691 30549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:36875 every 8 connection(s)
I20260923 17:33:48.432020 30483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:48.435807 30550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:48.440860 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 30483
I20260923 17:33:48.441030 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:48.441453 8311 external_mini_cluster.cc:1428] Setting key 741cf2277f8351cf12913575179fb652
I20260923 17:33:48.446609 30550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2: Bootstrap starting.
I20260923 17:33:48.450239 30550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:48.451444 30550 log.cc:824] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:48.454939 30550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2: No bootstrap required, opened a new log
I20260923 17:33:48.455097 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:48.443162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51513 (local address 127.8.29.254:36875)
0923 17:33:48.444007 (+ 845us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:48.444022 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:48.444067 (+ 45us) server_negotiation.cc:407] Connection header received
0923 17:33:48.445779 (+ 1712us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:48.445802 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:48.446208 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:48.446576 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:48.447101 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:48.448416 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:48.450195 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:48.451041 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:48.451183 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:48.452579 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:48.452590 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:48.452608 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:48.453313 (+ 705us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:48.453387 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:48.453412 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:48.453505 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:48.453799 (+ 294us) server_negotiation.cc:299] Negotiation successful
0923 17:33:48.454054 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":365,"threads_started":1}
I20260923 17:33:48.458716 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:48.464926 30550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aceb05ee3cf4240866f8135759cb4e2" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36875 } }
I20260923 17:33:48.465456 30550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:48.465575 30550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3aceb05ee3cf4240866f8135759cb4e2, State: Initialized, Role: FOLLOWER
I20260923 17:33:48.466190 30550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aceb05ee3cf4240866f8135759cb4e2" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36875 } }
I20260923 17:33:48.466436 30550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:48.466640 30550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:48.466840 30550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:48.470000 30550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aceb05ee3cf4240866f8135759cb4e2" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36875 } }
I20260923 17:33:48.470729 30550 leader_election.cc:302] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3aceb05ee3cf4240866f8135759cb4e2; no voters:
I20260923 17:33:48.470810 30553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:48.463866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60933 (local address 127.8.29.254:36875)
0923 17:33:48.464055 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:48.464060 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:48.464397 (+ 337us) server_negotiation.cc:407] Connection header received
0923 17:33:48.464608 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:48.464612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:48.464709 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:48.464785 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:48.465450 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:48.466314 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:48.468351 (+ 2037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:48.468841 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:48.468966 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:48.470083 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:48.470087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:48.470090 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:48.470108 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:48.470145 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:48.470159 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:48.470248 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:48.470388 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 17:33:48.470479 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260923 17:33:48.471401 30550 leader_election.cc:288] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:48.471493 30560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:48.472771 30560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [term 1 LEADER]: Becoming Leader. State: Replica: 3aceb05ee3cf4240866f8135759cb4e2, State: Running, Role: LEADER
I20260923 17:33:48.473285 30560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aceb05ee3cf4240866f8135759cb4e2" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36875 } }
I20260923 17:33:48.473590 30550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:48.476403 30562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3aceb05ee3cf4240866f8135759cb4e2. Latest consensus state: current_term: 1 leader_uuid: "3aceb05ee3cf4240866f8135759cb4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aceb05ee3cf4240866f8135759cb4e2" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36875 } } }
I20260923 17:33:48.476436 30561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3aceb05ee3cf4240866f8135759cb4e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3aceb05ee3cf4240866f8135759cb4e2" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 36875 } } }
I20260923 17:33:48.476881 30561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:48.476881 30562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:48.477632 30566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:48.481899 30566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:48.491667 30566 catalog_manager.cc:1347] Generated new cluster ID: d8af2995d01c40dca63009b64ba2711a
I20260923 17:33:48.491801 30566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:48.501900 30566 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:48.503273 30566 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:48.510927 30566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3aceb05ee3cf4240866f8135759cb4e2: Generated new TSK 0
I20260923 17:33:48.511796 30566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:48.520435 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30483
W20260923 17:33:48.521898 30545 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51513
W20260923 17:33:48.522843 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30483: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:48.523095 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 30483
2026-09-23T17:33:48Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1797 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:48 dist-test-slave-p7jt krb5kdc[30581](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:48 dist-test-slave-p7jt krb5kdc[30581](info): set up 2 sockets
Sep 23 17:33:48 dist-test-slave-p7jt krb5kdc[30581](info): commencing operation
krb5kdc: starting...
W20260923 17:33:49.582870 8311 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:49 dist-test-slave-p7jt krb5kdc[30581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:49.839260 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:37249
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:37249
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:50.171990 30594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:50.172319 30594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:50.172458 30594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:50.179723 30594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:50.179839 30594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:50.179903 30594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:50.179978 30594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:50.180037 30594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:50.190389 30594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:37249
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:37249
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:50.191708 30594 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:50.193212 30594 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:50.204830 30599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:50.204762 30600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:50.204799 30602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:50.205395 30594 server_base.cc:1034] running on GCE node
I20260923 17:33:50.206251 30594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:50.207732 30594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:50.208914 30594 hybrid_clock.cc:648] HybridClock initialized: now 1790184830208866 us; error 46 us; skew 500 ppm
I20260923 17:33:50.209241 30594 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:50 dist-test-slave-p7jt krb5kdc[30581](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184830, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:50.212252 30594 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:50.213727 30594 webserver.cc:460] Webserver started at http://127.8.29.254:36185/ using document root <none> and password file <none>
I20260923 17:33:50.214386 30594 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:50.214519 30594 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:50.214812 30594 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:50.217790 30594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "f2305390503f46c382961894b17b5972"
format_stamp: "Formatted at 2026-09-23 17:33:50 on dist-test-slave-p7jt"
server_key: "57c3178371fe8ccf6739240781e5dbba"
server_key_iv: "a43ac5795ae8095ed51d9eea2f122ec7"
server_key_version: "encryptionkey@0"
I20260923 17:33:50.218591 30594 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "f2305390503f46c382961894b17b5972"
format_stamp: "Formatted at 2026-09-23 17:33:50 on dist-test-slave-p7jt"
server_key: "57c3178371fe8ccf6739240781e5dbba"
server_key_iv: "a43ac5795ae8095ed51d9eea2f122ec7"
server_key_version: "encryptionkey@0"
I20260923 17:33:50.223815 30594 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260923 17:33:50.227381 30609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:50.228797 30594 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:50.228986 30594 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "f2305390503f46c382961894b17b5972"
format_stamp: "Formatted at 2026-09-23 17:33:50 on dist-test-slave-p7jt"
server_key: "57c3178371fe8ccf6739240781e5dbba"
server_key_iv: "a43ac5795ae8095ed51d9eea2f122ec7"
server_key_version: "encryptionkey@0"
I20260923 17:33:50.229175 30594 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:50.235101 30594 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:50.238960 30594 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:50.239238 30594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:50.255998 30594 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:37249
I20260923 17:33:50.256028 30660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:37249 every 8 connection(s)
I20260923 17:33:50.257369 30594 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:50.261063 30661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:50.264117 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 30594
I20260923 17:33:50.264264 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:50.264703 8311 external_mini_cluster.cc:1428] Setting key 7de93da95bd4a6e54d130e2dabcff190
I20260923 17:33:50.273351 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:50.266495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59907 (local address 127.8.29.254:37249)
0923 17:33:50.267289 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:50.267305 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:50.267352 (+ 47us) server_negotiation.cc:407] Connection header received
0923 17:33:50.269479 (+ 2127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:50.269509 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:50.269993 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:50.270436 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:50.271435 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:50.271443 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:50.271460 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:50.271483 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:50.271559 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:50.271586 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:50.271832 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:50.271956 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 17:33:50.272137 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":551,"thread_start_us":314,"threads_started":1}
I20260923 17:33:50.274087 30661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972: Bootstrap starting.
I20260923 17:33:50.277153 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:50.277689 30661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:50.278925 30661 log.cc:824] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:50.282311 30661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972: No bootstrap required, opened a new log
I20260923 17:33:50.284394 30664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:50.282067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43617 (local address 127.8.29.254:37249)
0923 17:33:50.282235 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:50.282241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:50.282590 (+ 349us) server_negotiation.cc:407] Connection header received
0923 17:33:50.282737 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:50.282742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:50.282813 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:50.282931 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:50.283761 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:50.283766 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:50.283770 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 17:33:50.283789 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:50.283838 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 17:33:50.283858 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:50.283996 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:50.284041 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 17:33:50.284103 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 17:33:50.290649 30661 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2305390503f46c382961894b17b5972" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37249 } }
I20260923 17:33:50.291091 30661 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 17:33:50.291219 30661 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2305390503f46c382961894b17b5972, State: Initialized, Role: FOLLOWER
I20260923 17:33:50.291841 30661 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2305390503f46c382961894b17b5972" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37249 } }
I20260923 17:33:50.292055 30661 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 17:33:50.292155 30661 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 17:33:50.292284 30661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:50.294384 30661 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2305390503f46c382961894b17b5972" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37249 } }
I20260923 17:33:50.294889 30661 leader_election.cc:302] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2305390503f46c382961894b17b5972; no voters:
I20260923 17:33:50.295415 30661 leader_election.cc:288] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 17:33:50.295542 30671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:50.296670 30671 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [term 1 LEADER]: Becoming Leader. State: Replica: f2305390503f46c382961894b17b5972, State: Running, Role: LEADER
I20260923 17:33:50.297199 30671 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2305390503f46c382961894b17b5972" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37249 } }
I20260923 17:33:50.297537 30661 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:50.299714 30672 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2305390503f46c382961894b17b5972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2305390503f46c382961894b17b5972" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37249 } } }
I20260923 17:33:50.300230 30672 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:50.299942 30673 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2305390503f46c382961894b17b5972. Latest consensus state: current_term: 1 leader_uuid: "f2305390503f46c382961894b17b5972" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2305390503f46c382961894b17b5972" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 37249 } } }
I20260923 17:33:50.300733 30673 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:50.302248 30678 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:50.307457 30678 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:50.316577 30678 catalog_manager.cc:1347] Generated new cluster ID: 0d4f485f69dd4a2d873de08a00160911
I20260923 17:33:50.316697 30678 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:50.331012 30678 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:50.331456 30678 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:50.337214 30678 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2305390503f46c382961894b17b5972: Generated new TSK 0
I20260923 17:33:50.338318 30678 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:50.350963 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30594
W20260923 17:33:50.352018 30656 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:59907
W20260923 17:33:50.352715 8311 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30594: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260923 17:33:50.352849 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 30594
2026-09-23T17:33:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1826 ms)
[----------] 18 tests from AuthTokenIssuingTest (33499 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:50 dist-test-slave-p7jt krb5kdc[30692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:50 dist-test-slave-p7jt krb5kdc[30692](info): set up 2 sockets
Sep 23 17:33:50 dist-test-slave-p7jt krb5kdc[30692](info): commencing operation
krb5kdc: starting...
W20260923 17:33:51.374969 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.989s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:51 dist-test-slave-p7jt krb5kdc[30692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:51.632558 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44643
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:44643,127.8.29.253:37549,127.8.29.252:40355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:51.965526 30705 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:33:51.965942 30705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:51.966059 30705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:51.966193 30705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:51.974021 30705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:51.974143 30705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:51.974236 30705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:51.974304 30705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:51.974392 30705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:51.984629 30705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:44643,127.8.29.253:37549,127.8.29.252:40355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:44643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:51.986022 30705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:51.987393 30705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:51.999004 30710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:51.999033 30711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:51.998971 30713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:51.999857 30705 server_base.cc:1034] running on GCE node
I20260923 17:33:52.000633 30705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:52.002090 30705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:52.003314 30705 hybrid_clock.cc:648] HybridClock initialized: now 1790184832003248 us; error 43 us; skew 500 ppm
I20260923 17:33:52.003625 30705 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:52 dist-test-slave-p7jt krb5kdc[30692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:52.012176 30705 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:52.013743 30705 webserver.cc:460] Webserver started at http://127.8.29.254:40901/ using document root <none> and password file <none>
I20260923 17:33:52.014520 30705 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:52.014642 30705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:52.014973 30705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:52.018038 30705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "0e9c60a86b7545cc97c364a9de068302"
format_stamp: "Formatted at 2026-09-23 17:33:52 on dist-test-slave-p7jt"
server_key: "837c2610403277756df93447b7b3f903"
server_key_iv: "c181daab7a5a26de8e75625f999dd796"
server_key_version: "encryptionkey@0"
I20260923 17:33:52.018879 30705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "0e9c60a86b7545cc97c364a9de068302"
format_stamp: "Formatted at 2026-09-23 17:33:52 on dist-test-slave-p7jt"
server_key: "837c2610403277756df93447b7b3f903"
server_key_iv: "c181daab7a5a26de8e75625f999dd796"
server_key_version: "encryptionkey@0"
I20260923 17:33:52.024283 30705 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260923 17:33:52.028085 30720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:52.029588 30705 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260923 17:33:52.029767 30705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "0e9c60a86b7545cc97c364a9de068302"
format_stamp: "Formatted at 2026-09-23 17:33:52 on dist-test-slave-p7jt"
server_key: "837c2610403277756df93447b7b3f903"
server_key_iv: "c181daab7a5a26de8e75625f999dd796"
server_key_version: "encryptionkey@0"
I20260923 17:33:52.030004 30705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:52.051608 30705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:52.056002 30705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:52.056265 30705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:52.074088 30705 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:44643
I20260923 17:33:52.074107 30771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:44643 every 8 connection(s)
I20260923 17:33:52.075528 30705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:52.077962 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 30705
I20260923 17:33:52.078194 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:52.079051 8311 external_mini_cluster.cc:1428] Setting key a9560c3a6a185d5f47d31e6d9d99d329
I20260923 17:33:52.080178 30772 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:52.089156 30772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:52 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.253@KRBTEST.COM" created.
I20260923 17:33:52.113533 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.092853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44643 (local address 127.0.0.1:59582)
0923 17:33:52.094073 (+ 1220us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.094119 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:52.096429 (+ 2310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:52.097371 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:52.097378 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:52.098105 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:52.098600 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:52.098629 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.100282 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.100287 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:52.101843 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:52.101858 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.102908 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.102918 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:52.103076 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:52.103751 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:52.103798 (+ 47us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:52.106370 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:52.109522 (+ 3152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:52.109535 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:52.109548 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:52.109964 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:52.110422 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:52.110426 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:52.110429 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:52.110622 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:52.111548 (+ 926us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:52.111554 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:52.111767 (+ 213us) client_negotiation.cc:769] Sending connection context
0923 17:33:52.111983 (+ 216us) client_negotiation.cc:240] Negotiation successful
0923 17:33:52.112415 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":948,"spinlock_wait_cycles":15744,"thread_start_us":714,"threads_started":1}
I20260923 17:33:52.113564 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.092880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59582 (local address 127.8.29.254:44643)
0923 17:33:52.093754 (+ 874us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:52.093769 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:52.094261 (+ 492us) server_negotiation.cc:407] Connection header received
0923 17:33:52.096865 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:52.096871 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:52.097121 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:52.097294 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:52.098849 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.100117 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.102000 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.102652 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.102800 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:52.106600 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:52.106608 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:52.106622 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:52.106655 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:52.109405 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:52.110091 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:52.110099 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:52.110102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:52.110285 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:52.110741 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:52.110745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:52.110748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:52.111410 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:52.111617 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:52.112156 (+ 539us) server_negotiation.cc:299] Negotiation successful
0923 17:33:52.112592 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":142,"server-negotiator.queue_time_us":618,"thread_start_us":373,"threads_started":1}
I20260923 17:33:52.116590 30772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:33:52.117700 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.117316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37210)
0923 17:33:52.117488 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.117589 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260923 17:33:52.118013 30722 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
I20260923 17:33:52.119557 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.119205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37218)
0923 17:33:52.119345 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.119413 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260923 17:33:52.119753 30772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37549: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab.
Entry for principal kudu/127.8.29.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab.
I20260923 17:33:52.142191 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.253:37549
--webserver_interface=127.8.29.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab
--principal=kudu/127.8.29.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:44643,127.8.29.253:37549,127.8.29.252:40355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:33:52.169108 30772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } attempt: 1
I20260923 17:33:52.170284 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.169831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37228)
0923 17:33:52.170034 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.170141 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 17:33:52.171326 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.170948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37236)
0923 17:33:52.171126 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.171202 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260923 17:33:52.171502 30772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37549: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
I20260923 17:33:52.239754 30772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } attempt: 2
I20260923 17:33:52.240931 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.240454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37240)
0923 17:33:52.240657 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.240786 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 17:33:52.241971 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.241565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37256)
0923 17:33:52.241731 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.241829 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260923 17:33:52.242184 30772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37549: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
I20260923 17:33:52.333410 30772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } attempt: 3
I20260923 17:33:52.334671 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.334130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37270)
0923 17:33:52.334405 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.334526 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260923 17:33:52.335695 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.335352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37274)
0923 17:33:52.335485 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.335555 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260923 17:33:52.335894 30772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37549: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
W20260923 17:33:52.475646 30779 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:33:52.476015 30779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:52.476131 30779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:52.476240 30779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260923 17:33:52.479130 30772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } attempt: 4
I20260923 17:33:52.480450 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.479847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37280)
0923 17:33:52.480182 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.480315 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260923 17:33:52.481520 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.481162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37282)
0923 17:33:52.481313 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.481398 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260923 17:33:52.481721 30772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:37549: Network error: Client connection negotiation failed: client connection to 127.8.29.253:37549: connect: Connection refused (error 111)
W20260923 17:33:52.484556 30779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:52.484655 30779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:52.484753 30779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:52.484823 30779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:52.484913 30779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:52.495671 30779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:44643,127.8.29.253:37549,127.8.29.252:40355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab
--principal=kudu/127.8.29.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.253:37549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.29.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:52.496982 30779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:52.498296 30779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:52.510214 30784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:52.510236 30787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:52.510264 30785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:52.510666 30779 server_base.cc:1034] running on GCE node
I20260923 17:33:52.511788 30779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:52.513301 30779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:52.514549 30779 hybrid_clock.cc:648] HybridClock initialized: now 1790184832514478 us; error 43 us; skew 500 ppm
I20260923 17:33:52.514891 30779 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:52 dist-test-slave-p7jt krb5kdc[30692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:52.518190 30779 init.cc:377] Logged in from keytab as kudu/127.8.29.253@KRBTEST.COM (short username kudu)
I20260923 17:33:52.520895 30779 webserver.cc:460] Webserver started at http://127.8.29.253:36567/ using document root <none> and password file <none>
I20260923 17:33:52.521821 30779 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:52.521951 30779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:52.522398 30779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:52.526485 30779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/data/instance:
uuid: "7085cf1642124ed5a843034ffe78b535"
format_stamp: "Formatted at 2026-09-23 17:33:52 on dist-test-slave-p7jt"
server_key: "f4d24859d7219f92b60fbb60cbe1fa5e"
server_key_iv: "43d9ffa8e11b70c81ea1deaeebc9acd4"
server_key_version: "encryptionkey@0"
I20260923 17:33:52.527554 30779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/wal/instance:
uuid: "7085cf1642124ed5a843034ffe78b535"
format_stamp: "Formatted at 2026-09-23 17:33:52 on dist-test-slave-p7jt"
server_key: "f4d24859d7219f92b60fbb60cbe1fa5e"
server_key_iv: "43d9ffa8e11b70c81ea1deaeebc9acd4"
server_key_version: "encryptionkey@0"
I20260923 17:33:52.533245 30779 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260923 17:33:52.537326 30794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:52.539103 30779 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260923 17:33:52.539299 30779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/wal
uuid: "7085cf1642124ed5a843034ffe78b535"
format_stamp: "Formatted at 2026-09-23 17:33:52 on dist-test-slave-p7jt"
server_key: "f4d24859d7219f92b60fbb60cbe1fa5e"
server_key_iv: "43d9ffa8e11b70c81ea1deaeebc9acd4"
server_key_version: "encryptionkey@0"
I20260923 17:33:52.539513 30779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:52.562920 30779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:52.567201 30779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:52.567459 30779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:52.585402 30779 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.253:37549
I20260923 17:33:52.585421 30845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.253:37549 every 8 connection(s)
I20260923 17:33:52.586788 30779 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
I20260923 17:33:52.590593 30846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:52.597168 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 30779
I20260923 17:33:52.597334 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-1/wal/instance
I20260923 17:33:52.597769 8311 external_mini_cluster.cc:1428] Setting key def86273fd0bb5b89c25914ae1cbd074
I20260923 17:33:52.598222 30846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:52 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:52.620293 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.600855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.8.29.254:44643)
0923 17:33:52.601052 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:52.601058 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:52.602376 (+ 1318us) server_negotiation.cc:407] Connection header received
0923 17:33:52.604835 (+ 2459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:52.604840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:52.604911 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:52.605001 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:52.606970 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.608040 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.609582 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.610162 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.610295 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:52.613993 (+ 3698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:52.613999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:52.614004 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:52.614046 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:52.617048 (+ 3002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:52.617869 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:52.617873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:52.617876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:52.618007 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:52.618431 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:52.618435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:52.618437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:52.618835 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:52.618974 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:52.619734 (+ 760us) server_negotiation.cc:299] Negotiation successful
0923 17:33:52.619976 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:33:52.621302 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.601362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44643 (local address 127.0.0.1:59584)
0923 17:33:52.602185 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.602240 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:52.604504 (+ 2264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:52.605282 (+ 778us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:52.605293 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:52.605974 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:52.606768 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:52.606793 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.608232 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.608237 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:52.609436 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:52.609448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.610409 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.610414 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:52.610697 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:52.611325 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:52.611352 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:52.613729 (+ 2377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:52.617227 (+ 3498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:52.617234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:52.617250 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:52.617730 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:52.618115 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:52.618120 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:52.618122 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:52.618312 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:52.618964 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:52.618973 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:52.619345 (+ 372us) client_negotiation.cc:769] Sending connection context
0923 17:33:52.619593 (+ 248us) client_negotiation.cc:240] Negotiation successful
0923 17:33:52.620143 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":320,"threads_started":1}
WARNING: no policy specified for kudu/127.8.29.252@KRBTEST.COM; defaulting to no policy
I20260923 17:33:52.623124 30846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.252@KRBTEST.COM" created.
Sep 23 17:33:52 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:33:52.638365 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.623928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37286)
0923 17:33:52.624098 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.624118 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:52.624248 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:52.625058 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:52.625068 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:52.625515 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:52.625683 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:52.625694 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.627219 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.627224 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:52.628337 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:52.628351 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.629234 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.629243 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:52.629416 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:52.629993 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:52.630026 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:52.632441 (+ 2415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:52.636169 (+ 3728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:52.636173 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:52.636176 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:52.636540 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:52.636911 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:52.636914 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:52.636916 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:52.637044 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:52.637674 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:52.637678 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:52.637793 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 17:33:52.637896 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 17:33:52.638093 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260923 17:33:52.638715 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.623944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37286 (local address 127.8.29.253:37549)
0923 17:33:52.624501 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:52.624510 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:52.624546 (+ 36us) server_negotiation.cc:407] Connection header received
0923 17:33:52.624603 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:52.624611 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:52.624872 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:52.625005 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:52.625845 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.627083 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.628484 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.629082 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.629183 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:52.632680 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:52.632722 (+ 42us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:52.632737 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:52.632773 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:52.636039 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:52.636661 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:52.636670 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:52.636672 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:52.636822 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:52.637138 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:52.637142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:52.637145 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:52.637557 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:52.637758 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:52.638066 (+ 308us) server_negotiation.cc:299] Negotiation successful
0923 17:33:52.638306 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":257,"threads_started":1}
I20260923 17:33:52.641207 30846 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:33:52.642218 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.641809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55142)
0923 17:33:52.641969 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.642081 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260923 17:33:52.642434 30798 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
I20260923 17:33:52.643963 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.643605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55156)
0923 17:33:52.643767 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.643862 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 17:33:52.644158 30846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:40355: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab.
Entry for principal kudu/127.8.29.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab.
I20260923 17:33:52.657873 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.252:40355
--webserver_interface=127.8.29.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab
--principal=kudu/127.8.29.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:33513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:44643,127.8.29.253:37549,127.8.29.252:40355
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:33:52.693521 30846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } attempt: 1
I20260923 17:33:52.694664 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.694165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55162)
0923 17:33:52.694378 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.694512 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 17:33:52.695576 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.695245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55164)
0923 17:33:52.695403 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.695474 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260923 17:33:52.695762 30846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:40355: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
I20260923 17:33:52.763999 30846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } attempt: 2
I20260923 17:33:52.765147 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.764687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55172)
0923 17:33:52.764907 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.764998 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260923 17:33:52.766086 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.765751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55186)
0923 17:33:52.765903 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.765972 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260923 17:33:52.766286 30846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:40355: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
I20260923 17:33:52.780984 30772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } attempt: 5
Sep 23 17:33:52 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:33:52.796240 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.781714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37302)
0923 17:33:52.781927 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.781948 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:52.782058 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:52.782442 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:52.782445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:52.782868 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:52.783046 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:52.783056 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.784460 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.784464 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:52.785788 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:52.785806 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.786731 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.786735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:52.786905 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:52.787465 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:52.787483 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:52.789472 (+ 1989us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:52.793665 (+ 4193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:52.793670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:52.793673 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:52.794055 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:52.794520 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:52.794525 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:52.794530 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:52.794721 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:52.795434 (+ 713us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:52.795438 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:52.795579 (+ 141us) client_negotiation.cc:769] Sending connection context
0923 17:33:52.795721 (+ 142us) client_negotiation.cc:240] Negotiation successful
0923 17:33:52.795944 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260923 17:33:52.796577 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.781831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37302 (local address 127.8.29.253:37549)
0923 17:33:52.782049 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:52.782055 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:52.782079 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:33:52.782145 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:52.782150 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:52.782267 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:52.782382 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:52.783221 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.784319 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.785959 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:52.786597 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:52.786719 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:52.789700 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:52.789707 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:52.789712 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:52.789762 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:52.793511 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:52.794174 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:52.794180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:52.794183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:52.794349 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:52.794881 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:52.794886 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:52.794889 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:52.795297 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:52.795457 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:52.795851 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 17:33:52.796079 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260923 17:33:52.797573 30772 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:33:52.798555 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.798150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55190)
0923 17:33:52.798313 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.798384 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260923 17:33:52.799455 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.799155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55200)
0923 17:33:52.799277 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.799354 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:33:52.799659 30772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:40355: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
I20260923 17:33:52.850868 30772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } attempt: 1
I20260923 17:33:52.851960 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.851495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55202)
0923 17:33:52.851698 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.851786 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260923 17:33:52.853040 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.852712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55206)
0923 17:33:52.852855 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.852930 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:33:52.853251 30772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:40355: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
I20260923 17:33:52.857510 30846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } attempt: 3
I20260923 17:33:52.858713 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.858187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55220)
0923 17:33:52.858407 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.858539 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260923 17:33:52.859853 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.859396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55228)
0923 17:33:52.859643 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.859725 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
W20260923 17:33:52.860069 30846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:40355: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
I20260923 17:33:52.921478 30772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } attempt: 2
I20260923 17:33:52.922752 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.922220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55244)
0923 17:33:52.922453 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.922589 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260923 17:33:52.923785 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:52.923440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55252)
0923 17:33:52.923592 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:52.923663 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:33:52.924001 30772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:40355: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
W20260923 17:33:52.985867 30853 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:33:52.986238 30853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:52.986340 30853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:52.986495 30853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:52.993806 30853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:52.993916 30853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:52.993997 30853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:52.994069 30853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:52.994153 30853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:53.003278 30846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } attempt: 4
I20260923 17:33:53.004410 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.003906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55258)
0923 17:33:53.004149 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:53.004239 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260923 17:33:53.004382 30853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:33513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:44643,127.8.29.253:37549,127.8.29.252:40355
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab
--principal=kudu/127.8.29.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.252:40355
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.29.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:53.005472 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.005114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55264)
0923 17:33:53.005280 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:53.005357 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260923 17:33:53.005681 30846 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:40355: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
I20260923 17:33:53.005900 30853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:53.007225 30853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:53.018694 30859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:53.018841 30858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:53.018936 30861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:53.019519 30853 server_base.cc:1034] running on GCE node
I20260923 17:33:53.020308 30853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:53.021678 30853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:53.022909 30853 hybrid_clock.cc:648] HybridClock initialized: now 1790184833022833 us; error 49 us; skew 500 ppm
I20260923 17:33:53.023236 30853 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:53 dist-test-slave-p7jt krb5kdc[30692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184833, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:53.026386 30853 init.cc:377] Logged in from keytab as kudu/127.8.29.252@KRBTEST.COM (short username kudu)
I20260923 17:33:53.027967 30853 webserver.cc:460] Webserver started at http://127.8.29.252:43713/ using document root <none> and password file <none>
I20260923 17:33:53.028682 30853 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:53.028790 30853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:53.029106 30853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:53.030237 30772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } attempt: 3
I20260923 17:33:53.031309 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.030839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55268)
0923 17:33:53.031046 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:53.031135 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 17:33:53.032472 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.032043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55270)
0923 17:33:53.032260 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:53.032334 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260923 17:33:53.032692 30772 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:40355: Network error: Client connection negotiation failed: client connection to 127.8.29.252:40355: connect: Connection refused (error 111)
I20260923 17:33:53.032548 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/data/instance:
uuid: "12f0bf1a0d45413a91b3deb70fbf032c"
format_stamp: "Formatted at 2026-09-23 17:33:53 on dist-test-slave-p7jt"
server_key: "e86c99f3c9fc3e1149a5f1bc646cb73b"
server_key_iv: "e18a5078930841f301b03731bbe7b91e"
server_key_version: "encryptionkey@0"
I20260923 17:33:53.033452 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/wal/instance:
uuid: "12f0bf1a0d45413a91b3deb70fbf032c"
format_stamp: "Formatted at 2026-09-23 17:33:53 on dist-test-slave-p7jt"
server_key: "e86c99f3c9fc3e1149a5f1bc646cb73b"
server_key_iv: "e18a5078930841f301b03731bbe7b91e"
server_key_version: "encryptionkey@0"
I20260923 17:33:53.038900 30853 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:33:53.042640 30868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:53.044498 30853 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:53.044653 30853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/wal
uuid: "12f0bf1a0d45413a91b3deb70fbf032c"
format_stamp: "Formatted at 2026-09-23 17:33:53 on dist-test-slave-p7jt"
server_key: "e86c99f3c9fc3e1149a5f1bc646cb73b"
server_key_iv: "e18a5078930841f301b03731bbe7b91e"
server_key_version: "encryptionkey@0"
I20260923 17:33:53.044850 30853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:53.057698 30853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:53.061856 30853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:53.062079 30853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:53.079653 30853 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.252:40355
I20260923 17:33:53.079685 30919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.252:40355 every 8 connection(s)
I20260923 17:33:53.081056 30853 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
I20260923 17:33:53.081360 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 30853
I20260923 17:33:53.081534 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790184478467572-8311-0/minicluster-data/master-2/wal/instance
I20260923 17:33:53.082131 8311 external_mini_cluster.cc:1428] Setting key c246b3d9e3d6143b638fdb964e469d11
I20260923 17:33:53.085736 30920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:53.092680 30920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:53 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
Sep 23 17:33:53 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184833, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:53.105135 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.084702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.8.29.252:40355)
0923 17:33:53.085508 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.085520 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.085563 (+ 43us) server_negotiation.cc:407] Connection header received
0923 17:33:53.087251 (+ 1688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.087274 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.087661 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.088021 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.089444 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.090876 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.092154 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.092835 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.093016 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.097506 (+ 4490us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.097514 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.097524 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.097561 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.100653 (+ 3092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.101415 (+ 762us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.101423 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.101430 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.101603 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.102022 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.102025 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.102027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.102654 (+ 627us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.102888 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.103435 (+ 547us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.103899 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":368,"threads_started":1}
I20260923 17:33:53.110826 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.095552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:44643 (local address 127.0.0.1:59586)
0923 17:33:53.096374 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:53.096416 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:53.096580 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:53.097014 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:53.097020 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:53.097662 (+ 642us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:53.097976 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.097987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.099179 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.099183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.100338 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.100354 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.101231 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.101237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.101449 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.102036 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:53.102107 (+ 71us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:53.104477 (+ 2370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:53.108007 (+ 3530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.108020 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.108027 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.108573 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.108996 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.109001 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.109005 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.109174 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.109699 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:53.109707 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:53.109933 (+ 226us) client_negotiation.cc:769] Sending connection context
0923 17:33:53.110153 (+ 220us) client_negotiation.cc:240] Negotiation successful
0923 17:33:53.110515 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"spinlock_wait_cycles":17152,"thread_start_us":412,"threads_started":1}
I20260923 17:33:53.110911 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.095633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59586 (local address 127.8.29.254:44643)
0923 17:33:53.095850 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.095857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.096518 (+ 661us) server_negotiation.cc:407] Connection header received
0923 17:33:53.096722 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.096728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.096813 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.096923 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.098202 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.099032 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.100500 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.101045 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.101143 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.104685 (+ 3542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.104693 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.104697 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.104742 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.107872 (+ 3130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.108720 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.108726 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.108730 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.108886 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.109288 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.109293 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.109296 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.109568 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.109700 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.110262 (+ 562us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.110436 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:33:53.112272 30920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:53 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
Sep 23 17:33:53 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184833, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:33:53.124593 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.108288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59596 (local address 127.8.29.254:44643)
0923 17:33:53.108890 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.108895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.108915 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:33:53.108956 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.108960 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.109012 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.109092 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.110580 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.111796 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.113237 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.113808 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.113945 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.118324 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.118332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.118337 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.118380 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.121487 (+ 3107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.122248 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.122254 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.122258 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.122443 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.122926 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.122931 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.122936 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.123341 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.123615 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.123919 (+ 304us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.124171 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":288,"threads_started":1}
I20260923 17:33:53.127527 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.113009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:37549 (local address 127.0.0.1:37308)
0923 17:33:53.113195 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:53.113213 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:53.113303 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:53.113582 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:53.113587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:53.114048 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:53.114209 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.114218 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.115559 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.115563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.116457 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.116468 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.117414 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.117421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.117632 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.118246 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:53.118273 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:53.120705 (+ 2432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:53.124550 (+ 3845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.124554 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.124557 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.125087 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.125561 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.125566 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.125569 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.125785 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.126551 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:53.126556 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:53.126734 (+ 178us) client_negotiation.cc:769] Sending connection context
0923 17:33:53.126853 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 17:33:53.127074 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 17:33:53.127602 30850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.113175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.8.29.253:37549)
0923 17:33:53.113336 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.113341 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.113356 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:33:53.113404 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.113408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.113466 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.113540 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.114367 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.115394 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.116598 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.117214 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.117343 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.120923 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.120929 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.120932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.120968 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.124385 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.125241 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.125247 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.125251 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.125421 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.125911 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.125916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.125920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.126198 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.126361 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.126991 (+ 630us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.127212 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 17:33:53.128655 30920 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:53 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
Sep 23 17:33:53 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184833, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
I20260923 17:33:53.143217 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.130054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55300)
0923 17:33:53.130229 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:53.130249 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:53.130365 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:53.130735 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:53.130740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:53.131232 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:53.131380 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.131389 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.132578 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.132584 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.133964 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.133982 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.134771 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.134778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.134999 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.135592 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:53.135608 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:53.137598 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:53.140851 (+ 3253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.140857 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.140861 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.141200 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.141570 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.141575 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.141579 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.141701 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.142182 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:53.142186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:53.142325 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 17:33:53.142448 (+ 123us) client_negotiation.cc:240] Negotiation successful
0923 17:33:53.142809 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":9}
I20260923 17:33:53.143352 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.130238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55300 (local address 127.8.29.252:40355)
0923 17:33:53.130365 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.130371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.130388 (+ 17us) server_negotiation.cc:407] Connection header received
0923 17:33:53.130545 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.130551 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.130615 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.130691 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.131517 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.132440 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.134140 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.134608 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.134758 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.137785 (+ 3027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.137791 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.137796 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.137832 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.140731 (+ 2899us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.141321 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.141327 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.141331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.141454 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.141802 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.141807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.141811 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.142066 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.142184 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.142560 (+ 376us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.142974 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":167,"server-negotiator.queue_time_us":34}
I20260923 17:33:53.143420 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.126327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37320 (local address 127.8.29.253:37549)
0923 17:33:53.126967 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.126972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.126992 (+ 20us) server_negotiation.cc:407] Connection header received
0923 17:33:53.127032 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.127037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.127097 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.127208 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.128670 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.129497 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.131240 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.131836 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.131966 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.136903 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.136908 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.136912 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.136949 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.140181 (+ 3232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.141066 (+ 885us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.141071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.141073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.141252 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.141700 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.141704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.141706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.142022 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.142294 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.142682 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.142923 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":325,"threads_started":1}
I20260923 17:33:53.148603 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:53.149227 30920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c: Bootstrap starting.
I20260923 17:33:53.152640 30920 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:53.153759 30920 log.cc:824] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:53.151233 30747 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0e9c60a86b7545cc97c364a9de068302"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59596
I20260923 17:33:53.156705 30920 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c: No bootstrap required, opened a new log
I20260923 17:33:53.164207 30920 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } }
I20260923 17:33:53.164564 30920 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12f0bf1a0d45413a91b3deb70fbf032c, State: Initialized, Role: FOLLOWER
I20260923 17:33:53.165030 30920 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } }
I20260923 17:33:53.165990 30928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.166375 30928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:53.166958 30920 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:33:53.176244 30939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:33:53.176390 30939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:33:53.174247 30853 master_runner.cc:186] Error getting master registration for 127.8.29.254:44643: OK, instance_id { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" instance_seqno: 1790184832068858 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:33:53.182199 30853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:33:53.209905 30772 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } attempt: 4
Sep 23 17:33:53 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
I20260923 17:33:53.229327 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.210809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55306 (local address 127.8.29.252:40355)
0923 17:33:53.211031 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.211037 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.211061 (+ 24us) server_negotiation.cc:407] Connection header received
0923 17:33:53.211181 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.211187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.211272 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.211383 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.212135 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.213070 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.214957 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.215745 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.215882 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.222891 (+ 7009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.222896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.222900 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.222937 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.226435 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.227168 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.227174 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.227178 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.227347 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.227712 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.227717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.227721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.228081 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.228287 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.228714 (+ 427us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.228953 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:33:53.229354 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.210723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55306)
0923 17:33:53.210914 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:53.210935 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:53.211044 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:53.211399 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:53.211402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:53.211758 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:53.211968 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.211979 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.213241 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.213247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.214786 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.214807 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.215943 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.215950 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.216166 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.219732 (+ 3566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:53.219763 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:53.222652 (+ 2889us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:53.226617 (+ 3965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.226624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.226629 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.227038 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.227454 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.227460 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.227464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.227604 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.228226 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:53.228232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:53.228434 (+ 202us) client_negotiation.cc:769] Sending connection context
0923 17:33:53.228609 (+ 175us) client_negotiation.cc:240] Negotiation successful
0923 17:33:53.228936 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 17:33:53.234246 30772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302: Bootstrap starting.
I20260923 17:33:53.238412 30772 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:53.239768 30772 log.cc:824] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:53.242925 30772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302: No bootstrap required, opened a new log
I20260923 17:33:53.251021 30772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } }
I20260923 17:33:53.251480 30772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e9c60a86b7545cc97c364a9de068302, State: Initialized, Role: FOLLOWER
I20260923 17:33:53.251962 30772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } }
I20260923 17:33:53.252890 30942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.253314 30942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:53.254120 30772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:53.254261 30747 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:33:53.254498 30747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:53.257997 30747 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } }
I20260923 17:33:53.260843 30747 leader_election.cc:288] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [CANDIDATE]: Term 1 election: Requested vote from peers 7085cf1642124ed5a843034ffe78b535 (127.8.29.253:37549), 12f0bf1a0d45413a91b3deb70fbf032c (127.8.29.252:40355)
I20260923 17:33:53.260978 30895 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e9c60a86b7545cc97c364a9de068302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "12f0bf1a0d45413a91b3deb70fbf032c"
I20260923 17:33:53.261631 30895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:53.265142 30895 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e9c60a86b7545cc97c364a9de068302 in term 1.
I20260923 17:33:53.265861 30724 leader_election.cc:302] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e9c60a86b7545cc97c364a9de068302, 12f0bf1a0d45413a91b3deb70fbf032c; no voters:
I20260923 17:33:53.266378 30942 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:53.276958 30942 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [term 1 LEADER]: Becoming Leader. State: Replica: 0e9c60a86b7545cc97c364a9de068302, State: Running, Role: LEADER
I20260923 17:33:53.277887 30942 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } }
I20260923 17:33:53.304883 30846 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } attempt: 5
W20260923 17:33:53.309331 30726 tablet.cc:2369] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:33:53.314149 30874 tablet.cc:2369] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:53.323177 30951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e9c60a86b7545cc97c364a9de068302. Latest consensus state: current_term: 1 leader_uuid: "0e9c60a86b7545cc97c364a9de068302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.323765 30951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:53.332391 30926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.269499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37324 (local address 127.8.29.253:37549)
0923 17:33:53.269800 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.269808 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.287992 (+ 18184us) server_negotiation.cc:407] Connection header received
0923 17:33:53.288874 (+ 882us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.288882 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.289094 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.289261 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.299844 (+ 10583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.300942 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.311297 (+ 10355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.312005 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.312115 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.324076 (+ 11961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.324084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.324090 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.324143 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.327888 (+ 3745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.328825 (+ 937us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.328853 (+ 28us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.328857 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.329085 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.329593 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.329599 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.329602 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.330166 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.330552 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.330783 (+ 231us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.331026 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 17:33:53.339496 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.298083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55318 (local address 127.8.29.252:40355)
0923 17:33:53.298307 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.298314 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.308482 (+ 10168us) server_negotiation.cc:407] Connection header received
0923 17:33:53.308903 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.308910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.309015 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.309095 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.312713 (+ 3618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.313782 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.318431 (+ 4649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.319181 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.319282 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.330848 (+ 11566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.330856 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.330862 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.330909 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.335141 (+ 4232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.336028 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.336036 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.336040 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.336227 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.336897 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.336903 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.336906 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.337291 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.337557 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.338181 (+ 624us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.338621 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 17:33:53.340873 30925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.276388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.8.29.254:44643)
0923 17:33:53.276634 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.276640 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.291549 (+ 14909us) server_negotiation.cc:407] Connection header received
0923 17:33:53.292669 (+ 1120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.292677 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.292827 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.292942 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.303019 (+ 10077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.304144 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.326604 (+ 22460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.327313 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.327443 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.330815 (+ 3372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.330822 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.330826 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.330866 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.334677 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.336639 (+ 1962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.336645 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.336649 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.336825 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.338867 (+ 2042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.338872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.338875 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.339253 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.339518 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.340178 (+ 660us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.340476 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 17:33:53.344125 30962 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:53.361604 30962 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 17:33:53 dist-test-slave-p7jt krb5kdc[30692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
I20260923 17:33:53.375548 30895 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [term 1 FOLLOWER]: Refusing update from remote peer 0e9c60a86b7545cc97c364a9de068302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:53.377911 30942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:53.370708 30705 master_runner.cc:186] Error getting master registration for 127.8.29.253:37549: OK, instance_id { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" instance_seqno: 1790184832580344 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:33:53.393628 30848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.305901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:40355 (local address 127.0.0.1:55332)
0923 17:33:53.306183 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:53.306204 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:53.306314 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:53.344287 (+ 37973us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:53.344293 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:53.344853 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:53.345092 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.345107 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.347170 (+ 2063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.347177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.348640 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:53.348659 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.358766 (+ 10107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.358775 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:53.359028 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.359789 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:53.359816 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:53.362800 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:53.390505 (+ 27705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.390512 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.390517 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.391146 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.391672 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:53.391677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:53.391681 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:53.391857 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:53.392581 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:53.392586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:53.392754 (+ 168us) client_negotiation.cc:769] Sending connection context
0923 17:33:53.392919 (+ 165us) client_negotiation.cc:240] Negotiation successful
0923 17:33:53.393238 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260923 17:33:53.395012 30960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:53.305980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55332 (local address 127.8.29.252:40355)
0923 17:33:53.339514 (+ 33534us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.339520 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.343909 (+ 4389us) server_negotiation.cc:407] Connection header received
0923 17:33:53.343973 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.343979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.344073 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.345270 (+ 1197us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.345541 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.346985 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.348866 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:53.349581 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:53.361495 (+ 11914us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:53.363126 (+ 1631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:53.363136 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:53.363140 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:53.363236 (+ 96us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:53.366723 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.391352 (+ 24629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.391358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.391362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.391530 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:53.391999 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:53.392004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:53.392007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:53.392442 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:53.394188 (+ 1746us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:53.394337 (+ 149us) server_negotiation.cc:299] Negotiation successful
0923 17:33:53.394611 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33386,"thread_start_us":35915,"threads_started":1}
I20260923 17:33:53.400975 30846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535: Bootstrap starting.
I20260923 17:33:53.404071 30928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e9c60a86b7545cc97c364a9de068302. Latest consensus state: current_term: 1 leader_uuid: "0e9c60a86b7545cc97c364a9de068302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.404450 30928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:53.405323 30846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:53.406636 30846 log.cc:824] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:53.407081 30705 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:33:53.425077 30928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e9c60a86b7545cc97c364a9de068302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.425367 30928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:53.427508 30951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e9c60a86b7545cc97c364a9de068302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.427814 30951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:53.431205 30962 catalog_manager.cc:1347] Generated new cluster ID: 832065526fa84715bab8f8370dd4f95a
I20260923 17:33:53.431327 30962 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:53.436129 30846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535: No bootstrap required, opened a new log
I20260923 17:33:53.449851 30846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } }
I20260923 17:33:53.450403 30846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7085cf1642124ed5a843034ffe78b535, State: Initialized, Role: FOLLOWER
I20260923 17:33:53.453125 30846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } }
I20260923 17:33:53.455577 30962 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:53.457535 30962 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:53.475099 30972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.475539 30972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:53.477440 30846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:53.477521 30821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0e9c60a86b7545cc97c364a9de068302" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7085cf1642124ed5a843034ffe78b535"
I20260923 17:33:53.478039 30821 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:53.481902 30821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e9c60a86b7545cc97c364a9de068302 in term 1.
I20260923 17:33:53.482836 30820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [term 1 FOLLOWER]: Refusing update from remote peer 0e9c60a86b7545cc97c364a9de068302: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:53.484753 30942 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:53.497293 30962 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302: Generated new TSK 0
I20260923 17:33:53.498451 30962 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:53.543380 30972 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e9c60a86b7545cc97c364a9de068302. Latest consensus state: current_term: 1 leader_uuid: "0e9c60a86b7545cc97c364a9de068302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.543736 30972 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:53.544528 30983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0e9c60a86b7545cc97c364a9de068302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.544952 30983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:53.547705 30951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0e9c60a86b7545cc97c364a9de068302" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0e9c60a86b7545cc97c364a9de068302" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 44643 } } peers { permanent_uuid: "7085cf1642124ed5a843034ffe78b535" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 37549 } } peers { permanent_uuid: "12f0bf1a0d45413a91b3deb70fbf032c" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 40355 } } }
I20260923 17:33:53.547977 30951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e9c60a86b7545cc97c364a9de068302 [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:53.555739 30989 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260923 17:33:53.629482 30925 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.616959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59614 (local address 127.8.29.254:44643)
0923 17:33:53.617391 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.617398 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.627481 (+ 10083us) server_negotiation.cc:407] Connection header received
0923 17:33:53.627775 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.627782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.627868 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.627946 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.629196 (+ 1250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59614: BlockingRecv error: recv got EOF from 127.0.0.1:59614 (error 108)
Metrics: {"server-negotiator.queue_time_us":295}
W20260923 17:33:53.631789 30960 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.627739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55344 (local address 127.8.29.252:40355)
0923 17:33:53.627933 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.627939 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.630097 (+ 2158us) server_negotiation.cc:407] Connection header received
0923 17:33:53.630350 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.630356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.630502 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.630574 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.631519 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55344: BlockingRecv error: recv got EOF from 127.0.0.1:55344 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260923 17:33:53.634227 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.617165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37328 (local address 127.8.29.253:37549)
0923 17:33:53.617389 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.617397 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.627192 (+ 9795us) server_negotiation.cc:407] Connection header received
0923 17:33:53.632631 (+ 5439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.632638 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.632723 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.632788 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.633988 (+ 1200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37328: BlockingRecv error: recv got EOF from 127.0.0.1:37328 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:33:53.645166 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.639633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37344 (local address 127.8.29.253:37549)
0923 17:33:53.639801 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.639807 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.643056 (+ 3249us) server_negotiation.cc:407] Connection header received
0923 17:33:53.643432 (+ 376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.643439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.643602 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.643716 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.644928 (+ 1212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37344: BlockingRecv error: recv got EOF from 127.0.0.1:37344 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260923 17:33:53.652545 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.650130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.8.29.253:37549)
0923 17:33:53.650320 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.650326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.650988 (+ 662us) server_negotiation.cc:407] Connection header received
0923 17:33:53.651161 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.651165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.651254 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.651440 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.652278 (+ 838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37356: BlockingRecv error: recv got EOF from 127.0.0.1:37356 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 17:33:53.660449 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.658145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37362 (local address 127.8.29.253:37549)
0923 17:33:53.658351 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.658358 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.658887 (+ 529us) server_negotiation.cc:407] Connection header received
0923 17:33:53.659086 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.659093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.659198 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.659307 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.660239 (+ 932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37362: BlockingRecv error: recv got EOF from 127.0.0.1:37362 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:33:53.670739 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.668399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37366 (local address 127.8.29.253:37549)
0923 17:33:53.668638 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.668645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.669163 (+ 518us) server_negotiation.cc:407] Connection header received
0923 17:33:53.669355 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.669361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.669470 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.669595 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.670432 (+ 837us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37366: BlockingRecv error: recv got EOF from 127.0.0.1:37366 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
W20260923 17:33:53.684873 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.682545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.8.29.253:37549)
0923 17:33:53.682738 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.682745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.683297 (+ 552us) server_negotiation.cc:407] Connection header received
0923 17:33:53.683523 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.683529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.683621 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.683737 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.684638 (+ 901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37372: BlockingRecv error: recv got EOF from 127.0.0.1:37372 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
W20260923 17:33:53.706776 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.704535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37378 (local address 127.8.29.253:37549)
0923 17:33:53.704731 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.704737 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.705252 (+ 515us) server_negotiation.cc:407] Connection header received
0923 17:33:53.705462 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.705469 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.705580 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.705704 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.706525 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37378: BlockingRecv error: recv got EOF from 127.0.0.1:37378 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 17:33:53.744598 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.742732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.8.29.253:37549)
0923 17:33:53.742989 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.742995 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.743255 (+ 260us) server_negotiation.cc:407] Connection header received
0923 17:33:53.743441 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.743447 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.743547 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.743674 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.744361 (+ 687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37380: BlockingRecv error: recv got EOF from 127.0.0.1:37380 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260923 17:33:53.819603 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.817619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.8.29.253:37549)
0923 17:33:53.817809 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.817815 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.818207 (+ 392us) server_negotiation.cc:407] Connection header received
0923 17:33:53.818409 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.818413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.818522 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.818612 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.819370 (+ 758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37392: BlockingRecv error: recv got EOF from 127.0.0.1:37392 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260923 17:33:53.953506 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:53.951613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37394 (local address 127.8.29.253:37549)
0923 17:33:53.951860 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:53.951867 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:53.952167 (+ 300us) server_negotiation.cc:407] Connection header received
0923 17:33:53.952395 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:53.952404 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:53.952477 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:53.952561 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:53.953342 (+ 781us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37394: BlockingRecv error: recv got EOF from 127.0.0.1:37394 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260923 17:33:54.178660 30939 catalog_manager.cc:1259] Loaded cluster ID: 832065526fa84715bab8f8370dd4f95a
I20260923 17:33:54.178773 30939 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c: loading cluster ID for follower catalog manager: success
I20260923 17:33:54.181311 30939 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c: acquiring CA information for follower catalog manager: success
I20260923 17:33:54.183188 30939 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12f0bf1a0d45413a91b3deb70fbf032c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:33:54.214936 30926 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:54.213214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37410 (local address 127.8.29.253:37549)
0923 17:33:54.213468 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:54.213475 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:54.213588 (+ 113us) server_negotiation.cc:407] Connection header received
0923 17:33:54.213803 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:54.213808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:54.213879 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:54.213965 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:54.214730 (+ 765us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37410: BlockingRecv error: recv got EOF from 127.0.0.1:37410 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260923 17:33:54.560662 30989 catalog_manager.cc:1259] Loaded cluster ID: 832065526fa84715bab8f8370dd4f95a
I20260923 17:33:54.560817 30989 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535: loading cluster ID for follower catalog manager: success
I20260923 17:33:54.564635 30989 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535: acquiring CA information for follower catalog manager: success
I20260923 17:33:54.567269 30989 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7085cf1642124ed5a843034ffe78b535: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:33:54.738097 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:54.730795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37424 (local address 127.8.29.253:37549)
0923 17:33:54.731428 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:54.731435 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:54.731727 (+ 292us) server_negotiation.cc:407] Connection header received
0923 17:33:54.731976 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:54.731981 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:54.732144 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:54.732282 (+ 138us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:54.732790 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.733646 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.735706 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.736240 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.736342 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:54.736907 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:54.737499 (+ 592us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:54.737609 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:54.737741 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 17:33:54.737811 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":297,"threads_started":1}
I20260923 17:33:54.750988 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:54.744684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55350 (local address 127.8.29.252:40355)
0923 17:33:54.745246 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:54.745250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:54.745283 (+ 33us) server_negotiation.cc:407] Connection header received
0923 17:33:54.745373 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:54.745377 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:54.745531 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:54.745661 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:54.746097 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.746850 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.748717 (+ 1867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.749113 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.749214 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:54.749810 (+ 596us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:54.750345 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:54.750451 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:54.750606 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 17:33:54.750676 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":346,"threads_started":1}
I20260923 17:33:54.770129 31054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:54.758260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37432 (local address 127.8.29.253:37549)
0923 17:33:54.758477 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:54.758485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:54.760282 (+ 1797us) server_negotiation.cc:407] Connection header received
0923 17:33:54.761299 (+ 1017us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:54.761306 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:54.761402 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:54.761512 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:54.762634 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.763934 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.767342 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.768212 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.768353 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:54.769136 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:54.769315 (+ 179us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:54.769389 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:54.769656 (+ 267us) server_negotiation.cc:299] Negotiation successful
0923 17:33:54.769773 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:33:54.774636 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:54.757992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.8.29.254:44643)
0923 17:33:54.766940 (+ 8948us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:54.766948 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:54.767327 (+ 379us) server_negotiation.cc:407] Connection header received
0923 17:33:54.767460 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:54.767466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:54.767678 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:54.768524 (+ 846us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:54.768691 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.769891 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.772036 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.772677 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.773293 (+ 616us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:54.773489 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:54.773946 (+ 457us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:54.774058 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:54.774219 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 17:33:54.774308 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8787,"thread_start_us":8462,"threads_started":1}
I20260923 17:33:54.777041 31060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:54.761042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.8.29.252:40355)
0923 17:33:54.761228 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:54.761234 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:54.771267 (+ 10033us) server_negotiation.cc:407] Connection header received
0923 17:33:54.771431 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:54.771436 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:54.771537 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:54.771621 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:54.772171 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.773180 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.774831 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:54.775389 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:54.775514 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:54.776235 (+ 721us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:54.776380 (+ 145us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:54.776444 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:54.776692 (+ 248us) server_negotiation.cc:299] Negotiation successful
0923 17:33:54.776778 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260923 17:33:54.780131 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30705
I20260923 17:33:54.781136 30767 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:54.892915 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 30705
I20260923 17:33:54.919726 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30779
I20260923 17:33:54.920943 30841 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:55.029711 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 30779
I20260923 17:33:55.046329 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 30853
I20260923 17:33:55.047523 30915 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:55.154817 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 30853
2026-09-23T17:33:55Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4810 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 17:33:55 dist-test-slave-p7jt krb5kdc[31074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 17:33:55 dist-test-slave-p7jt krb5kdc[31074](info): set up 2 sockets
Sep 23 17:33:55 dist-test-slave-p7jt krb5kdc[31074](info): commencing operation
krb5kdc: starting...
W20260923 17:33:56.190802 8311 mini_kdc.cc:121] Time spent starting KDC: real 0.995s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/test-user.keytab.
Sep 23 17:33:56 dist-test-slave-p7jt krb5kdc[31074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T17:33:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T17:33:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.29.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
Entry for principal kudu/127.8.29.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab.
I20260923 17:33:56.442380 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:38117
--webserver_interface=127.8.29.254
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:38117,127.8.29.253:34693,127.8.29.252:38887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 17:33:56.773485 31087 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:33:56.773828 31087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:56.773926 31087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:56.774032 31087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:56.781417 31087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:56.781525 31087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:56.781586 31087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:56.781654 31087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:56.781744 31087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:56.791903 31087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:38117,127.8.29.253:34693,127.8.29.252:38887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.254.keytab
--principal=kudu/127.8.29.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.254:38117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.29.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:56.793241 31087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:56.794596 31087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:56.806844 31093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:56.807071 31092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:56.807382 31095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:56.808125 31087 server_base.cc:1034] running on GCE node
I20260923 17:33:56.808910 31087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:56.810379 31087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:56.811681 31087 hybrid_clock.cc:648] HybridClock initialized: now 1790184836811556 us; error 62 us; skew 500 ppm
I20260923 17:33:56.812031 31087 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:56 dist-test-slave-p7jt krb5kdc[31074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184836, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:56.815227 31087 init.cc:377] Logged in from keytab as kudu/127.8.29.254@KRBTEST.COM (short username kudu)
I20260923 17:33:56.816763 31087 webserver.cc:460] Webserver started at http://127.8.29.254:43331/ using document root <none> and password file <none>
I20260923 17:33:56.817447 31087 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:56.817548 31087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:56.817862 31087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:56.820961 31087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/data/instance:
uuid: "3a5cadab5f6c41a68c2476bb4a463f9c"
format_stamp: "Formatted at 2026-09-23 17:33:56 on dist-test-slave-p7jt"
server_key: "57dfdefbf1dd6a7367bba1714ff32aea"
server_key_iv: "66b46a4f8716b9bb01f76a726f1c947e"
server_key_version: "encryptionkey@0"
I20260923 17:33:56.821767 31087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/wal/instance:
uuid: "3a5cadab5f6c41a68c2476bb4a463f9c"
format_stamp: "Formatted at 2026-09-23 17:33:56 on dist-test-slave-p7jt"
server_key: "57dfdefbf1dd6a7367bba1714ff32aea"
server_key_iv: "66b46a4f8716b9bb01f76a726f1c947e"
server_key_version: "encryptionkey@0"
I20260923 17:33:56.827333 31087 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260923 17:33:56.830945 31102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:56.832523 31087 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260923 17:33:56.832705 31087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/wal
uuid: "3a5cadab5f6c41a68c2476bb4a463f9c"
format_stamp: "Formatted at 2026-09-23 17:33:56 on dist-test-slave-p7jt"
server_key: "57dfdefbf1dd6a7367bba1714ff32aea"
server_key_iv: "66b46a4f8716b9bb01f76a726f1c947e"
server_key_version: "encryptionkey@0"
I20260923 17:33:56.832957 31087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:56.850946 31087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:56.855384 31087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:56.855639 31087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:56.873520 31087 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.254:38117
I20260923 17:33:56.873540 31153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.254:38117 every 8 connection(s)
I20260923 17:33:56.874922 31087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/data/info.pb
I20260923 17:33:56.877578 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 31087
I20260923 17:33:56.877769 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-0/wal/instance
I20260923 17:33:56.878319 8311 external_mini_cluster.cc:1428] Setting key 7df5f4d1dbf740594d918b5b65d900c0
I20260923 17:33:56.879124 31154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:56.887985 31154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:56 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184836, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.253@KRBTEST.COM" created.
I20260923 17:33:56.911684 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:56.891813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.8.29.254:38117)
0923 17:33:56.892670 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:56.892688 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:56.893210 (+ 522us) server_negotiation.cc:407] Connection header received
0923 17:33:56.895418 (+ 2208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:56.895423 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:56.895702 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:56.895888 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:56.897285 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:56.898566 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:56.899956 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:56.900536 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:56.900711 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:56.904124 (+ 3413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:56.904133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:56.904148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:56.904177 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:56.907263 (+ 3086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:56.907978 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:56.907985 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:56.907987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:56.908137 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:56.908512 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:56.908516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:56.908518 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:56.909052 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:56.909246 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:56.909806 (+ 560us) server_negotiation.cc:299] Negotiation successful
0923 17:33:56.910148 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":618,"thread_start_us":382,"threads_started":1}
I20260923 17:33:56.911705 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:56.891814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38117 (local address 127.0.0.1:52838)
0923 17:33:56.893044 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:56.893085 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:56.895006 (+ 1921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:56.895934 (+ 928us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:56.895941 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:56.896668 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:56.897032 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:56.897052 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:56.898709 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:56.898712 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:56.899822 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:56.899835 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:56.900781 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:56.900788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:56.901018 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:56.901624 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:56.901649 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:56.903892 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:56.907407 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:56.907420 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:56.907435 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:56.907846 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:56.908245 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:56.908250 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:56.908254 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:56.908408 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:56.909186 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:56.909196 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:56.909424 (+ 228us) client_negotiation.cc:769] Sending connection context
0923 17:33:56.909653 (+ 229us) client_negotiation.cc:240] Negotiation successful
0923 17:33:56.910296 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1008,"mutex_wait_us":170,"thread_start_us":813,"threads_started":1}
I20260923 17:33:56.915344 31154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:33:56.916498 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:56.916073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56436)
0923 17:33:56.916266 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:56.916375 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260923 17:33:56.916757 31106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
I20260923 17:33:56.918318 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:56.917966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56444)
0923 17:33:56.918109 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:56.918179 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260923 17:33:56.918543 31154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:34693: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab.
Entry for principal kudu/127.8.29.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab.
I20260923 17:33:56.939000 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.253:34693
--webserver_interface=127.8.29.253
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab
--principal=kudu/127.8.29.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:38117,127.8.29.253:34693,127.8.29.252:38887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:33:56.967907 31154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } attempt: 1
I20260923 17:33:56.969185 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:56.968672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56458)
0923 17:33:56.968915 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:56.969026 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260923 17:33:56.970271 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:56.969894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56470)
0923 17:33:56.970055 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:56.970126 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:33:56.970495 31154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:34693: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
I20260923 17:33:57.038750 31154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } attempt: 2
I20260923 17:33:57.039942 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.039437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56472)
0923 17:33:57.039675 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.039800 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 17:33:57.040910 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.040556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56484)
0923 17:33:57.040693 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.040780 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:33:57.041095 31154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:34693: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
I20260923 17:33:57.132380 31154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } attempt: 3
I20260923 17:33:57.133512 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.133079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56490)
0923 17:33:57.133282 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.133375 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260923 17:33:57.134500 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.134106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56504)
0923 17:33:57.134252 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.134320 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260923 17:33:57.134660 31154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:34693: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
W20260923 17:33:57.265770 31161 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:33:57.266146 31161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:57.266242 31161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:57.266360 31161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:57.274319 31161 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:57.274420 31161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:57.274529 31161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:57.274600 31161 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:57.274683 31161 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:57.277909 31154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } attempt: 4
I20260923 17:33:57.279053 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.278520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56510)
0923 17:33:57.278842 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.278926 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182}
I20260923 17:33:57.280009 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.279666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56522)
0923 17:33:57.279808 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.279889 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260923 17:33:57.280172 31154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.253:34693: Network error: Client connection negotiation failed: client connection to 127.8.29.253:34693: connect: Connection refused (error 111)
I20260923 17:33:57.285205 31161 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:38117,127.8.29.253:34693,127.8.29.252:38887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.253.keytab
--principal=kudu/127.8.29.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.253:34693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.29.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:57.286595 31161 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:57.287906 31161 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 17:33:57.299685 31166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:57.299618 31169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:57.299649 31167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:57.300402 31161 server_base.cc:1034] running on GCE node
I20260923 17:33:57.301144 31161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:57.302573 31161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:57.303798 31161 hybrid_clock.cc:648] HybridClock initialized: now 1790184837303727 us; error 66 us; skew 500 ppm
I20260923 17:33:57.304126 31161 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:57.314829 31161 init.cc:377] Logged in from keytab as kudu/127.8.29.253@KRBTEST.COM (short username kudu)
I20260923 17:33:57.316432 31161 webserver.cc:460] Webserver started at http://127.8.29.253:45315/ using document root <none> and password file <none>
I20260923 17:33:57.317169 31161 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:57.317284 31161 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:57.317592 31161 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:57.320688 31161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/data/instance:
uuid: "3b62ab55a5d5458297d6a87aac92a51a"
format_stamp: "Formatted at 2026-09-23 17:33:57 on dist-test-slave-p7jt"
server_key: "4906a07d162fb6a6e3df98cf674578b7"
server_key_iv: "77a2ec7d0fbe0fb056d18ae2c778bc32"
server_key_version: "encryptionkey@0"
I20260923 17:33:57.321475 31161 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/wal/instance:
uuid: "3b62ab55a5d5458297d6a87aac92a51a"
format_stamp: "Formatted at 2026-09-23 17:33:57 on dist-test-slave-p7jt"
server_key: "4906a07d162fb6a6e3df98cf674578b7"
server_key_iv: "77a2ec7d0fbe0fb056d18ae2c778bc32"
server_key_version: "encryptionkey@0"
I20260923 17:33:57.326876 31161 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260923 17:33:57.330531 31176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:57.332088 31161 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:57.332284 31161 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/wal
uuid: "3b62ab55a5d5458297d6a87aac92a51a"
format_stamp: "Formatted at 2026-09-23 17:33:57 on dist-test-slave-p7jt"
server_key: "4906a07d162fb6a6e3df98cf674578b7"
server_key_iv: "77a2ec7d0fbe0fb056d18ae2c778bc32"
server_key_version: "encryptionkey@0"
I20260923 17:33:57.332531 31161 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:57.347872 31161 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:57.352321 31161 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:57.352600 31161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:57.370482 31161 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.253:34693
I20260923 17:33:57.370519 31227 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.253:34693 every 8 connection(s)
I20260923 17:33:57.371852 31161 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/data/info.pb
I20260923 17:33:57.373873 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 31161
I20260923 17:33:57.374056 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-1/wal/instance
I20260923 17:33:57.374647 8311 external_mini_cluster.cc:1428] Setting key 632c8a573c059c8cc9f5b2e54d6f529d
I20260923 17:33:57.376241 31228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:57.385070 31228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.29.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.29.252@KRBTEST.COM" created.
I20260923 17:33:57.406145 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.388054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52848 (local address 127.8.29.254:38117)
0923 17:33:57.388221 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:57.388225 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:57.389632 (+ 1407us) server_negotiation.cc:407] Connection header received
0923 17:33:57.391600 (+ 1968us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:57.391606 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:57.391687 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:57.391772 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:57.393324 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.394001 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.395640 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.396104 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.396232 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.399945 (+ 3713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:57.399951 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:57.399955 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:57.399992 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:57.402566 (+ 2574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.403530 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.403535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.403537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.403719 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.404252 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.404257 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.404259 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.404566 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:57.404707 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:57.405690 (+ 983us) server_negotiation.cc:299] Negotiation successful
0923 17:33:57.405900 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260923 17:33:57.407362 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.388663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38117 (local address 127.0.0.1:52848)
0923 17:33:57.389460 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.389498 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:57.391319 (+ 1821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:57.392020 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:57.392029 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:57.392581 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:57.393147 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.393166 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.394154 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.394158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.395485 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.395498 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.396343 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.396350 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.396652 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.397387 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:57.397416 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:57.399672 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:57.402720 (+ 3048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.402730 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.402758 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.403363 (+ 605us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.403831 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.403836 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.403840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.404089 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.404700 (+ 611us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:57.404708 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:57.405180 (+ 472us) client_negotiation.cc:769] Sending connection context
0923 17:33:57.405481 (+ 301us) client_negotiation.cc:240] Negotiation successful
0923 17:33:57.406009 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":341,"threads_started":1}
I20260923 17:33:57.409257 31228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:33:57.425385 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.410129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56528)
0923 17:33:57.410380 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.410396 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:57.410509 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:57.411387 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:57.411392 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:57.411829 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:57.411997 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.412008 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.414060 (+ 2052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.414064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.415001 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.415014 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.416108 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.416113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.416238 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.416824 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:57.416842 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:57.419076 (+ 2234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:57.422492 (+ 3416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.422497 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.422501 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.423026 (+ 525us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.423452 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.423457 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.423460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.423627 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.424346 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:57.424351 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:57.424500 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 17:33:57.424650 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 17:33:57.424951 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139}
I20260923 17:33:57.425730 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.410211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.8.29.253:34693)
0923 17:33:57.410937 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:57.410944 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:57.410972 (+ 28us) server_negotiation.cc:407] Connection header received
0923 17:33:57.411020 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:57.411026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:57.411228 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:57.411353 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:57.412202 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.413904 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.415137 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.415941 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.416061 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.419315 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:57.419328 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:57.419344 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:57.419379 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:57.422314 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.423207 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.423215 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.423218 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.423350 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.423783 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.423787 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.423790 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.424216 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:57.424390 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:57.424863 (+ 473us) server_negotiation.cc:299] Negotiation successful
0923 17:33:57.425113 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":413,"threads_started":1}
I20260923 17:33:57.428328 31228 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:33:57.429636 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.429025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46418)
0923 17:33:57.429235 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.429370 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260923 17:33:57.429939 31177 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
I20260923 17:33:57.431608 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.431186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46430)
0923 17:33:57.431351 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.431440 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 17:33:57.431816 31228 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:38887: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.29.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab.
Entry for principal kudu/127.8.29.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab.
I20260923 17:33:57.438704 8311 external_mini_cluster.cc:1326] Running /tmp/dist-test-task6Gj89p/build/asan/bin/kudu
/tmp/dist-test-task6Gj89p/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.252:38887
--webserver_interface=127.8.29.252
--webserver_port=0
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab
--principal=kudu/127.8.29.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.29.212:46043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.29.254:38117,127.8.29.253:34693,127.8.29.252:38887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 17:33:57.481159 31228 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } attempt: 1
I20260923 17:33:57.482542 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.481927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46436)
0923 17:33:57.482177 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.482314 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 17:33:57.483731 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.483331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46446)
0923 17:33:57.483500 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.483593 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260923 17:33:57.483968 31228 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:38887: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
I20260923 17:33:57.552256 31228 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } attempt: 2
I20260923 17:33:57.553498 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.552964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46462)
0923 17:33:57.553198 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.553311 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260923 17:33:57.554555 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.554141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46466)
0923 17:33:57.554287 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.554373 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:33:57.554771 31228 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:38887: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
I20260923 17:33:57.579434 31154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } attempt: 5
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184836, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:33:57.594449 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.580292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.8.29.253:34693)
0923 17:33:57.580458 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:57.580463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:57.580482 (+ 19us) server_negotiation.cc:407] Connection header received
0923 17:33:57.580672 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:57.580677 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:57.580784 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:57.580867 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:57.581931 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.582677 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.584366 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.584815 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.584909 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.588819 (+ 3910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:57.588825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:57.588828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:57.588865 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:57.591497 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.592360 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.592366 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.592369 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.592521 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.593012 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.593017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.593020 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.593333 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:57.593462 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:57.593985 (+ 523us) server_negotiation.cc:299] Negotiation successful
0923 17:33:57.594220 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:33:57.594547 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.580155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56544)
0923 17:33:57.580376 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.580399 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:57.580538 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:57.580937 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:57.580942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:57.581499 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:57.581750 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.581766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.582831 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.582836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.584211 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.584227 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.584993 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.585000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.585240 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.585867 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:57.585890 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:57.588550 (+ 2660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:57.591652 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.591658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.591663 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.592174 (+ 511us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.592695 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.592700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.592704 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.592871 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.593460 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:57.593465 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:57.593659 (+ 194us) client_negotiation.cc:769] Sending connection context
0923 17:33:57.593857 (+ 198us) client_negotiation.cc:240] Negotiation successful
0923 17:33:57.594179 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260923 17:33:57.595770 31154 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:33:57.597074 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.596577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46482)
0923 17:33:57.596778 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.596882 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260923 17:33:57.598137 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.597766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46498)
0923 17:33:57.597921 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.598016 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260923 17:33:57.598363 31154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:38887: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
I20260923 17:33:57.646027 31228 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } attempt: 3
I20260923 17:33:57.647399 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.646846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46506)
0923 17:33:57.647085 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.647211 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260923 17:33:57.648558 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.648074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46514)
0923 17:33:57.648329 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.648420 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260923 17:33:57.648806 31228 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:38887: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
I20260923 17:33:57.649588 31154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } attempt: 1
I20260923 17:33:57.650683 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.650154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46528)
0923 17:33:57.650354 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.650494 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260923 17:33:57.651718 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.651392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46544)
0923 17:33:57.651520 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.651594 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:33:57.651940 31154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:38887: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
I20260923 17:33:57.720167 31154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } attempt: 2
I20260923 17:33:57.721369 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.720866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46552)
0923 17:33:57.721111 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.721211 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260923 17:33:57.722445 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.722060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46554)
0923 17:33:57.722225 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.722303 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260923 17:33:57.722703 31154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:38887: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
W20260923 17:33:57.772079 31235 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 17:33:57.772435 31235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 17:33:57.772531 31235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 17:33:57.772626 31235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 17:33:57.780027 31235 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 17:33:57.780131 31235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 17:33:57.780193 31235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 17:33:57.780262 31235 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 17:33:57.780346 31235 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 17:33:57.790609 31235 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.29.212:46043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.29.254:38117,127.8.29.253:34693,127.8.29.252:38887
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/krb5kdc/kudu_127.8.29.252.keytab
--principal=kudu/127.8.29.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.29.252:38887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.29.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 23 Sep 2026 16:28:54 UTC on ccycloud.quasar-niuser.root.comops.site
build id 4667752
ASAN enabled
I20260923 17:33:57.792053 31235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 17:33:57.792059 31228 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } attempt: 4
I20260923 17:33:57.793273 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.792762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46560)
0923 17:33:57.793033 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.793121 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260923 17:33:57.793776 31235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 17:33:57.794373 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.794025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46570)
0923 17:33:57.794177 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.794254 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260923 17:33:57.794597 31228 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:38887: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
W20260923 17:33:57.805672 31243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:57.805730 31240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 17:33:57.805642 31241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 17:33:57.806918 31235 server_base.cc:1034] running on GCE node
I20260923 17:33:57.807899 31235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 17:33:57.810009 31235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 17:33:57.811297 31235 hybrid_clock.cc:648] HybridClock initialized: now 1790184837811222 us; error 51 us; skew 500 ppm
I20260923 17:33:57.811697 31235 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 17:33:57.814909 31235 init.cc:377] Logged in from keytab as kudu/127.8.29.252@KRBTEST.COM (short username kudu)
I20260923 17:33:57.816366 31235 webserver.cc:460] Webserver started at http://127.8.29.252:46169/ using document root <none> and password file <none>
I20260923 17:33:57.817045 31235 fs_manager.cc:359] Metadata directory not provided
I20260923 17:33:57.817137 31235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 17:33:57.817413 31235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 17:33:57.820520 31235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/data/instance:
uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f"
format_stamp: "Formatted at 2026-09-23 17:33:57 on dist-test-slave-p7jt"
server_key: "c619f1f0988c6962667be9b52ddabbeb"
server_key_iv: "1da610a3e9d7d6996b20126c487b002b"
server_key_version: "encryptionkey@0"
I20260923 17:33:57.821270 31235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/wal/instance:
uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f"
format_stamp: "Formatted at 2026-09-23 17:33:57 on dist-test-slave-p7jt"
server_key: "c619f1f0988c6962667be9b52ddabbeb"
server_key_iv: "1da610a3e9d7d6996b20126c487b002b"
server_key_version: "encryptionkey@0"
I20260923 17:33:57.826599 31235 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260923 17:33:57.828967 31154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } attempt: 3
I20260923 17:33:57.830045 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.829628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46574)
0923 17:33:57.829807 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.829893 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 17:33:57.830621 31250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 17:33:57.831080 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.830683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46582)
0923 17:33:57.830897 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.830971 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260923 17:33:57.831267 31154 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.29.252:38887: Network error: Client connection negotiation failed: client connection to 127.8.29.252:38887: connect: Connection refused (error 111)
I20260923 17:33:57.832087 31235 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260923 17:33:57.832254 31235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/data,/tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/wal
uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f"
format_stamp: "Formatted at 2026-09-23 17:33:57 on dist-test-slave-p7jt"
server_key: "c619f1f0988c6962667be9b52ddabbeb"
server_key_iv: "1da610a3e9d7d6996b20126c487b002b"
server_key_version: "encryptionkey@0"
I20260923 17:33:57.832482 31235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 17:33:57.848393 31235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 17:33:57.853230 31235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 17:33:57.853503 31235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 17:33:57.870704 31235 rpc_server.cc:307] RPC server started. Bound to: 127.8.29.252:38887
I20260923 17:33:57.870759 31301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.29.252:38887 every 8 connection(s)
I20260923 17:33:57.872154 31235 server_base.cc:1166] Dumped server information to /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/data/info.pb
I20260923 17:33:57.873201 8311 external_mini_cluster.cc:1388] Started /tmp/dist-test-task6Gj89p/build/asan/bin/kudu as pid 31235
I20260923 17:33:57.873413 8311 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task6Gj89p/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790184478467572-8311-0/minicluster-data/master-2/wal/instance
I20260923 17:33:57.873984 8311 external_mini_cluster.cc:1428] Setting key ec33dbdab2a643484c51c39f07f091c1
I20260923 17:33:57.877110 31302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 17:33:57.887145 31302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:57.905534 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.889922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52856 (local address 127.8.29.254:38117)
0923 17:33:57.890133 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:57.890139 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:57.890911 (+ 772us) server_negotiation.cc:407] Connection header received
0923 17:33:57.891112 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:57.891118 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:57.891224 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:57.891317 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:57.892734 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.893619 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.895487 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.895907 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.896021 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.899698 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:57.899703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:57.899706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:57.899749 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:57.902385 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.903252 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.903259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.903262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.903432 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.903929 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.903933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.903935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.904224 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:57.904362 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:57.904994 (+ 632us) server_negotiation.cc:299] Negotiation successful
0923 17:33:57.905225 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 17:33:57.906803 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.882186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.8.29.252:38887)
0923 17:33:57.884702 (+ 2516us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:57.884724 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:57.885533 (+ 809us) server_negotiation.cc:407] Connection header received
0923 17:33:57.887334 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:57.887359 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:57.887822 (+ 463us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:57.888195 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:57.889803 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.891507 (+ 1704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.893172 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.894114 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.894993 (+ 879us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.898781 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:57.898795 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:57.898814 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:57.898855 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:57.901826 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.902790 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.902799 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.902802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.902999 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.903528 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.903533 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.903536 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.904238 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:57.904539 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:57.904856 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 17:33:57.905301 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":2220,"thread_start_us":1934,"threads_started":1}
I20260923 17:33:57.906822 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.889933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.254:38117 (local address 127.0.0.1:52856)
0923 17:33:57.890767 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.890809 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:57.890969 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:57.891373 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:57.891378 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:57.892225 (+ 847us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:57.892558 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.892571 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.893789 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.893793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.895295 (+ 1502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.895311 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.896131 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.896139 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.896315 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.896881 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:57.896918 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:57.899500 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:57.902553 (+ 3053us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.902563 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.902573 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.903086 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.903531 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.903536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.903539 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.903773 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.904356 (+ 583us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:57.904365 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:57.904585 (+ 220us) client_negotiation.cc:769] Sending connection context
0923 17:33:57.904856 (+ 271us) client_negotiation.cc:240] Negotiation successful
0923 17:33:57.905439 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":571,"mutex_wait_us":127,"spinlock_wait_cycles":27392,"thread_start_us":373,"threads_started":1}
I20260923 17:33:57.908576 31302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.254@KRBTEST.COM
I20260923 17:33:57.923377 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.909368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56558 (local address 127.8.29.253:34693)
0923 17:33:57.909629 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:57.909634 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:57.909665 (+ 31us) server_negotiation.cc:407] Connection header received
0923 17:33:57.909820 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:57.909825 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:57.909887 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:57.909963 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:57.910856 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.911882 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.913596 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.914230 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.914365 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.917696 (+ 3331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:57.917703 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:57.917708 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:57.917752 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:57.920761 (+ 3009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.921433 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.921439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.921443 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.921606 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.921973 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.921978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.921981 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.922304 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:57.922492 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:57.922938 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 17:33:57.923105 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260923 17:33:57.923574 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.909421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.253:34693 (local address 127.0.0.1:56558)
0923 17:33:57.909592 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.909611 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:57.909700 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:57.910015 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:57.910019 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:57.910452 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:57.910674 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.910722 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.912037 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.912041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.913455 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.913467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.914400 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.914405 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.914620 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.915213 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:57.915237 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:57.917500 (+ 2263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:57.920893 (+ 3393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.920897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.920900 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.921302 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.921704 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.921707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.921710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.921872 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.922501 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:57.922508 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:57.922687 (+ 179us) client_negotiation.cc:769] Sending connection context
0923 17:33:57.922832 (+ 145us) client_negotiation.cc:240] Negotiation successful
0923 17:33:57.923179 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260923 17:33:57.924692 31302 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } has no permanent_uuid. Determining permanent_uuid...
I20260923 17:33:57.925777 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.910275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52866 (local address 127.8.29.254:38117)
0923 17:33:57.910439 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:57.910444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:57.910501 (+ 57us) server_negotiation.cc:407] Connection header received
0923 17:33:57.910558 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:57.910562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:57.910629 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:57.910726 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:57.912141 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.913251 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.914876 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.915439 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.915568 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.919713 (+ 4145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:57.919720 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:57.919725 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:57.919766 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:57.922763 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.923645 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.923651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.923654 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.923830 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.924235 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.924239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.924241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.924496 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:57.924755 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:57.925033 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 17:33:57.925266 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.252@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
Sep 23 17:33:57 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.29.253@KRBTEST.COM
I20260923 17:33:57.938494 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.925461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.8.29.252:38887)
0923 17:33:57.925630 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:57.925636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:57.925827 (+ 191us) server_negotiation.cc:407] Connection header received
0923 17:33:57.926035 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:57.926040 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:57.926115 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:57.926218 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:57.927145 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.927966 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.929090 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.929495 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.929604 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.933013 (+ 3409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:57.933018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:57.933021 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:57.933049 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:57.935583 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.936352 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.936358 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.936363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.936492 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.936951 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.936957 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.936961 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.937308 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:57.937441 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:57.937852 (+ 411us) server_negotiation.cc:299] Negotiation successful
0923 17:33:57.938065 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260923 17:33:57.938685 31306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.925515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46604)
0923 17:33:57.925747 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:57.925763 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:57.925865 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:57.926272 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:57.926277 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:57.926773 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:57.926972 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.926986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.928080 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.928085 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.928967 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:57.928979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.929620 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.929626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:57.929813 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.930325 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:57.930347 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:57.932841 (+ 2494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:57.935778 (+ 2937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.935784 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.935789 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.936243 (+ 454us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.936642 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:57.936646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:57.936650 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:57.936830 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:57.937429 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:57.937433 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:57.937609 (+ 176us) client_negotiation.cc:769] Sending connection context
0923 17:33:57.937764 (+ 155us) client_negotiation.cc:240] Negotiation successful
0923 17:33:57.938230 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":141}
I20260923 17:33:57.942301 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:57.927708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56568 (local address 127.8.29.253:34693)
0923 17:33:57.927895 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:57.927901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:57.927917 (+ 16us) server_negotiation.cc:407] Connection header received
0923 17:33:57.927962 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:57.927967 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:57.928028 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:57.928117 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:57.929269 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.929933 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.931624 (+ 1691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:57.932272 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:57.932413 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:57.936899 (+ 4486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:57.936904 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:57.936908 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:57.936944 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:57.939763 (+ 2819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.940436 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.940442 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.940446 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.940592 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:57.940974 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:57.940979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:57.940982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:57.941250 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:57.941513 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:57.941774 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 17:33:57.942002 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260923 17:33:57.944015 31302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f: Bootstrap starting.
I20260923 17:33:57.947410 8311 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 17:33:57.947621 31302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:57.948928 31302 log.cc:824] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:57.952410 31302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f: No bootstrap required, opened a new log
I20260923 17:33:57.950351 31129 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52866
I20260923 17:33:57.963402 31302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } }
I20260923 17:33:57.963864 31302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 883f8aa2e8ff4ed8bb6ed800e557ec6f, State: Initialized, Role: FOLLOWER
I20260923 17:33:57.964447 31302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } }
I20260923 17:33:57.965477 31308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:57.965854 31308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:57.966856 31302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [sys.catalog]: configured and running, proceeding with master startup.
W20260923 17:33:57.977072 31319 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:33:57.977243 31319 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:33:57.974635 31235 master_runner.cc:186] Error getting master registration for 127.8.29.254:38117: OK, instance_id { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" instance_seqno: 1790184836868476 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:33:57.984297 31235 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:33:58.008502 31154 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } attempt: 4
Sep 23 17:33:58 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184836, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.254@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
I20260923 17:33:58.025827 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:58.009385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.8.29.252:38887)
0923 17:33:58.009766 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.009771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.009786 (+ 15us) server_negotiation.cc:407] Connection header received
0923 17:33:58.009837 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.009841 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.009898 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.009971 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.011060 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.012238 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.014079 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.014729 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.014835 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:58.018811 (+ 3976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:58.018821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:58.018825 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:58.018859 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:58.022660 (+ 3801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.023567 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.023574 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.023578 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.023737 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.024160 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.024166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.024169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.024641 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:58.024830 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:58.025218 (+ 388us) server_negotiation.cc:299] Negotiation successful
0923 17:33:58.025454 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253}
I20260923 17:33:58.025825 31158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:58.009250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46614)
0923 17:33:58.009438 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:58.009458 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:58.009569 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:58.010069 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:58.010075 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:58.010659 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:58.010886 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:58.010901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.012437 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.012444 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:58.013909 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:58.013928 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.014887 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.014894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:58.015116 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:58.015808 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:58.015834 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:58.018605 (+ 2771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:58.022832 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:58.022837 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:58.022840 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:58.023413 (+ 573us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:58.023861 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:58.023866 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:58.023871 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:58.024050 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:58.024760 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:58.024764 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:58.024944 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 17:33:58.025088 (+ 144us) client_negotiation.cc:240] Negotiation successful
0923 17:33:58.025402 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260923 17:33:58.030413 31154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c: Bootstrap starting.
I20260923 17:33:58.033937 31154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:58.035286 31154 log.cc:824] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:58.041815 31154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c: No bootstrap required, opened a new log
I20260923 17:33:58.049719 31154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } }
I20260923 17:33:58.050184 31154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a5cadab5f6c41a68c2476bb4a463f9c, State: Initialized, Role: FOLLOWER
I20260923 17:33:58.050809 31154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } }
I20260923 17:33:58.052099 31322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:58.052744 31322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:58.053435 31154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:58.053511 31129 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 17:33:58.053676 31129 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:58.057502 31129 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } }
I20260923 17:33:58.061379 31277 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f"
I20260923 17:33:58.061931 31277 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:58.065572 31277 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5cadab5f6c41a68c2476bb4a463f9c in term 1.
I20260923 17:33:58.066272 31103 leader_election.cc:302] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a5cadab5f6c41a68c2476bb4a463f9c, 883f8aa2e8ff4ed8bb6ed800e557ec6f; no voters:
I20260923 17:33:58.066835 31129 leader_election.cc:288] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [CANDIDATE]: Term 1 election: Requested vote from peers 3b62ab55a5d5458297d6a87aac92a51a (127.8.29.253:34693), 883f8aa2e8ff4ed8bb6ed800e557ec6f (127.8.29.252:38887)
I20260923 17:33:58.073601 31322 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 17:33:58.063740 31087 master_runner.cc:186] Error getting master registration for 127.8.29.253:34693: OK, instance_id { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" instance_seqno: 1790184837365264 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 17:33:58.081007 31087 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 17:33:58.083520 31322 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [term 1 LEADER]: Becoming Leader. State: Replica: 3a5cadab5f6c41a68c2476bb4a463f9c, State: Running, Role: LEADER
W20260923 17:33:58.084357 31333 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 17:33:58.084457 31333 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 17:33:58.084227 31322 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } }
I20260923 17:33:58.093784 31228 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } attempt: 5
Sep 23 17:33:58 dist-test-slave-p7jt krb5kdc[31074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790184837, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.29.253@KRBTEST.COM for kudu/127.8.29.252@KRBTEST.COM
W20260923 17:33:58.105914 31256 tablet.cc:2369] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 17:33:58.108552 31108 tablet.cc:2369] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 17:33:58.115815 31231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:58.094568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.29.252:38887 (local address 127.0.0.1:46626)
0923 17:33:58.094877 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0923 17:33:58.094896 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0923 17:33:58.095033 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 17:33:58.096015 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 17:33:58.096020 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 17:33:58.096574 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 17:33:58.096815 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:58.096829 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.098532 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.098539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:58.100033 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 17:33:58.100051 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.101402 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.101409 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 17:33:58.101635 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:58.102497 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 17:33:58.102548 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0923 17:33:58.106754 (+ 4206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 17:33:58.111149 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:58.111156 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:58.111161 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:58.112051 (+ 890us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:58.112840 (+ 789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 17:33:58.112846 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 17:33:58.112850 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 17:33:58.113110 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 17:33:58.114164 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 17:33:58.114169 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 17:33:58.114416 (+ 247us) client_negotiation.cc:769] Sending connection context
0923 17:33:58.114650 (+ 234us) client_negotiation.cc:240] Negotiation successful
0923 17:33:58.115141 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199}
I20260923 17:33:58.117806 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:58.094591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46626 (local address 127.8.29.252:38887)
0923 17:33:58.094785 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.094796 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.094973 (+ 177us) server_negotiation.cc:407] Connection header received
0923 17:33:58.095523 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.095530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.095702 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.095808 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.097046 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.098312 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.100294 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.100992 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.101119 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:58.107058 (+ 5939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:58.107066 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:58.107072 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:58.107118 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:58.110933 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.112294 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.112300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.112305 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.112667 (+ 362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.113341 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.113348 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.113352 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.113963 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:58.116402 (+ 2439us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:58.116671 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 17:33:58.117158 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260923 17:33:58.124454 31228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a: Bootstrap starting.
I20260923 17:33:58.133260 31336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a5cadab5f6c41a68c2476bb4a463f9c. Latest consensus state: current_term: 1 leader_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:58.133837 31336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:58.141671 31228 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a: Neither blocks nor log segments found. Creating new log.
I20260923 17:33:58.145191 31340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 17:33:58.147467 31340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 17:33:58.149299 31228 log.cc:824] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a: Log is configured to *not* fsync() on all Append() calls
I20260923 17:33:58.157197 31228 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a: No bootstrap required, opened a new log
I20260923 17:33:58.157475 31277 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [term 1 FOLLOWER]: Refusing update from remote peer 3a5cadab5f6c41a68c2476bb4a463f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:58.158644 31336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:58.169009 31228 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } }
I20260923 17:33:58.169617 31228 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b62ab55a5d5458297d6a87aac92a51a, State: Initialized, Role: FOLLOWER
I20260923 17:33:58.170331 31228 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } }
I20260923 17:33:58.177615 31348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:58.179906 31203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3b62ab55a5d5458297d6a87aac92a51a"
I20260923 17:33:58.180299 31202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [term 0 FOLLOWER]: Advancing to term 1
I20260923 17:33:58.179875 31228 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 17:33:58.181264 31348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:58.184093 31202 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [term 1 FOLLOWER]: Refusing update from remote peer 3a5cadab5f6c41a68c2476bb4a463f9c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 17:33:58.185153 31336 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 17:33:58.187414 31203 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3a5cadab5f6c41a68c2476bb4a463f9c in term 1.
I20260923 17:33:58.209904 31348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a5cadab5f6c41a68c2476bb4a463f9c. Latest consensus state: current_term: 1 leader_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:58.210343 31348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:58.212659 31342 mvcc.cc:205] Tried to move back new op lower bound from 7332597097078968320 to 7332597096799883264. Current Snapshot: MvccSnapshot[applied={T|T < 7332597097078968320}]
I20260923 17:33:58.213706 31348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:58.214121 31348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:58.223769 31308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a5cadab5f6c41a68c2476bb4a463f9c. Latest consensus state: current_term: 1 leader_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:58.224121 31308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:58.228499 31336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:58.228793 31336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:58.233443 31350 mvcc.cc:205] Tried to move back new op lower bound from 7332597097078968320 to 7332597096799883264. Current Snapshot: MvccSnapshot[applied={T|T < 7332597097078968320}]
I20260923 17:33:58.235333 31322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:58.236330 31322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c [sys.catalog]: This master's current role is: LEADER
I20260923 17:33:58.238359 31347 mvcc.cc:205] Tried to move back new op lower bound from 7332597097078968320 to 7332597096799883264. Current Snapshot: MvccSnapshot[applied={T|T < 7332597097078968320}]
I20260923 17:33:58.241915 31340 catalog_manager.cc:1347] Generated new cluster ID: 448ad4234221496e94ad1c81b285b2db
I20260923 17:33:58.242039 31340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 17:33:58.250969 31308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a5cadab5f6c41a68c2476bb4a463f9c" member_type: VOTER last_known_addr { host: "127.8.29.254" port: 38117 } } peers { permanent_uuid: "3b62ab55a5d5458297d6a87aac92a51a" member_type: VOTER last_known_addr { host: "127.8.29.253" port: 34693 } } peers { permanent_uuid: "883f8aa2e8ff4ed8bb6ed800e557ec6f" member_type: VOTER last_known_addr { host: "127.8.29.252" port: 38887 } } }
I20260923 17:33:58.251284 31308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f [sys.catalog]: This master's current role is: FOLLOWER
I20260923 17:33:58.259677 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:58.164689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56576 (local address 127.8.29.253:34693)
0923 17:33:58.164859 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.164864 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.189492 (+ 24628us) server_negotiation.cc:407] Connection header received
0923 17:33:58.190952 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.190956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.191046 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.191114 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.206643 (+ 15529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.207802 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.217554 (+ 9752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.218281 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.218384 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:58.228356 (+ 9972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:58.228364 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:58.228368 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:58.228416 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:58.232128 (+ 3712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.247717 (+ 15589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.247724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.247728 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.247945 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.250052 (+ 2107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.250059 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.250062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.250507 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:58.250783 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:58.258849 (+ 8066us) server_negotiation.cc:299] Negotiation successful
0923 17:33:58.259218 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260923 17:33:58.261368 31157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:58.163830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52872 (local address 127.8.29.254:38117)
0923 17:33:58.164021 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.164027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.187555 (+ 23528us) server_negotiation.cc:407] Connection header received
0923 17:33:58.189900 (+ 2345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.189906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.189993 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.190063 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.208049 (+ 17986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.209112 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.215518 (+ 6406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.216304 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.216417 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:58.233685 (+ 17268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:58.233692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:58.233697 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:58.233768 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:58.238016 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.248575 (+ 10559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.248582 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.248585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.248817 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.252034 (+ 3217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.252040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.252043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.252517 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:58.252784 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:58.260540 (+ 7756us) server_negotiation.cc:299] Negotiation successful
0923 17:33:58.260895 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260923 17:33:58.267369 31305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:58.164048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46642 (local address 127.8.29.252:38887)
0923 17:33:58.164239 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.164245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.188991 (+ 24746us) server_negotiation.cc:407] Connection header received
0923 17:33:58.192946 (+ 3955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.192953 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.193046 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.193119 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.212668 (+ 19549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.213621 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.219449 (+ 5828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:58.220176 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:58.220299 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:58.239737 (+ 19438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 17:33:58.239746 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 17:33:58.239752 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 17:33:58.239811 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0923 17:33:58.243633 (+ 3822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.249006 (+ 5373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.249012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.249015 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.249191 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 17:33:58.251301 (+ 2110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 17:33:58.251307 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 17:33:58.251310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0923 17:33:58.251708 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 17:33:58.251933 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:58.266548 (+ 14615us) server_negotiation.cc:299] Negotiation successful
0923 17:33:58.266888 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260923 17:33:58.284544 31340 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 17:33:58.286572 31340 catalog_manager.cc:1504] Loading token signing keys...
I20260923 17:33:58.312872 31340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a5cadab5f6c41a68c2476bb4a463f9c: Generated new TSK 0
I20260923 17:33:58.314168 31340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 17:33:58.389045 31368 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260923 17:33:58.417436 31157 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.411375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52878 (local address 127.8.29.254:38117)
0923 17:33:58.411535 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.411542 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.415395 (+ 3853us) server_negotiation.cc:407] Connection header received
0923 17:33:58.415696 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.415702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.415783 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.415868 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.417185 (+ 1317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52878: BlockingRecv error: recv got EOF from 127.0.0.1:52878 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260923 17:33:58.418542 31305 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.411144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46648 (local address 127.8.29.252:38887)
0923 17:33:58.411325 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.411331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.415402 (+ 4071us) server_negotiation.cc:407] Connection header received
0923 17:33:58.415691 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.415697 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.415807 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.415893 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.418289 (+ 2396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46648: BlockingRecv error: recv got EOF from 127.0.0.1:46648 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260923 17:33:58.423751 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.411025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56584 (local address 127.8.29.253:34693)
0923 17:33:58.411222 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.411228 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.418965 (+ 7737us) server_negotiation.cc:407] Connection header received
0923 17:33:58.419157 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.419163 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.419259 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.419353 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.423517 (+ 4164us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56584: BlockingRecv error: recv got EOF from 127.0.0.1:56584 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 17:33:58.432556 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.428200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56586 (local address 127.8.29.253:34693)
0923 17:33:58.428373 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.428380 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.430995 (+ 2615us) server_negotiation.cc:407] Connection header received
0923 17:33:58.431200 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.431206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.431341 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.431459 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.432278 (+ 819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56586: BlockingRecv error: recv got EOF from 127.0.0.1:56586 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260923 17:33:58.444851 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.438706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.8.29.253:34693)
0923 17:33:58.438902 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.438908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.443222 (+ 4314us) server_negotiation.cc:407] Connection header received
0923 17:33:58.443432 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.443439 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.443545 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.443671 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.444619 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56598: BlockingRecv error: recv got EOF from 127.0.0.1:56598 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260923 17:33:58.455595 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.450106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.8.29.253:34693)
0923 17:33:58.450295 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.450301 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.453968 (+ 3667us) server_negotiation.cc:407] Connection header received
0923 17:33:58.454152 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.454158 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.454266 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.454392 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.455336 (+ 944us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56604: BlockingRecv error: recv got EOF from 127.0.0.1:56604 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20260923 17:33:58.465449 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.463146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.8.29.253:34693)
0923 17:33:58.463356 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.463362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.463928 (+ 566us) server_negotiation.cc:407] Connection header received
0923 17:33:58.464131 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.464137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.464243 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.464362 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.465214 (+ 852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56606: BlockingRecv error: recv got EOF from 127.0.0.1:56606 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
W20260923 17:33:58.479319 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.477020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.8.29.253:34693)
0923 17:33:58.477214 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.477220 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.477801 (+ 581us) server_negotiation.cc:407] Connection header received
0923 17:33:58.477998 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.478004 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.478105 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.478222 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.479080 (+ 858us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56614: BlockingRecv error: recv got EOF from 127.0.0.1:56614 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260923 17:33:58.502249 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.499823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56624 (local address 127.8.29.253:34693)
0923 17:33:58.500033 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.500039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.500612 (+ 573us) server_negotiation.cc:407] Connection header received
0923 17:33:58.500840 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.500846 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.500947 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.501070 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.501993 (+ 923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56624: BlockingRecv error: recv got EOF from 127.0.0.1:56624 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260923 17:33:58.541040 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.538296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56640 (local address 127.8.29.253:34693)
0923 17:33:58.538512 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.538519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.539462 (+ 943us) server_negotiation.cc:407] Connection header received
0923 17:33:58.539688 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.539694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.539796 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.539907 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.540777 (+ 870us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56640: BlockingRecv error: recv got EOF from 127.0.0.1:56640 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260923 17:33:58.611325 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.609260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.8.29.253:34693)
0923 17:33:58.609461 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.609468 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.609781 (+ 313us) server_negotiation.cc:407] Connection header received
0923 17:33:58.610007 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.610013 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.610111 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.610233 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.611086 (+ 853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56654: BlockingRecv error: recv got EOF from 127.0.0.1:56654 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260923 17:33:58.745641 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:58.743910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56656 (local address 127.8.29.253:34693)
0923 17:33:58.744106 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:58.744111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:58.744379 (+ 268us) server_negotiation.cc:407] Connection header received
0923 17:33:58.744571 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:58.744576 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:58.744654 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:58.744738 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:58.745434 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56656: BlockingRecv error: recv got EOF from 127.0.0.1:56656 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
I20260923 17:33:58.979400 31319 catalog_manager.cc:1259] Loaded cluster ID: 448ad4234221496e94ad1c81b285b2db
I20260923 17:33:58.979511 31319 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f: loading cluster ID for follower catalog manager: success
I20260923 17:33:58.982036 31319 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f: acquiring CA information for follower catalog manager: success
I20260923 17:33:58.983898 31319 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 883f8aa2e8ff4ed8bb6ed800e557ec6f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260923 17:33:59.008064 31232 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 17:33:59.006242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56662 (local address 127.8.29.253:34693)
0923 17:33:59.006536 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:59.006542 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:59.006770 (+ 228us) server_negotiation.cc:407] Connection header received
0923 17:33:59.006953 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:59.006958 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:59.007041 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:59.007137 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 17:33:59.007840 (+ 703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56662: BlockingRecv error: recv got EOF from 127.0.0.1:56662 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
I20260923 17:33:59.393651 31368 catalog_manager.cc:1259] Loaded cluster ID: 448ad4234221496e94ad1c81b285b2db
I20260923 17:33:59.393759 31368 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a: loading cluster ID for follower catalog manager: success
I20260923 17:33:59.396354 31368 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a: acquiring CA information for follower catalog manager: success
I20260923 17:33:59.398188 31368 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3b62ab55a5d5458297d6a87aac92a51a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 17:33:59.532696 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:59.524989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.8.29.253:34693)
0923 17:33:59.525664 (+ 675us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:59.525671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:59.526005 (+ 334us) server_negotiation.cc:407] Connection header received
0923 17:33:59.526234 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:59.526238 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:59.526407 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:59.526561 (+ 154us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:59.527023 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.527824 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.529832 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.530379 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.530521 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:59.531270 (+ 749us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:59.531896 (+ 626us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:59.532033 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:59.532230 (+ 197us) server_negotiation.cc:299] Negotiation successful
0923 17:33:59.532327 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":314,"threads_started":1}
I20260923 17:33:59.560201 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:59.539921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46658 (local address 127.8.29.252:38887)
0923 17:33:59.553622 (+ 13701us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:59.553627 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:59.553650 (+ 23us) server_negotiation.cc:407] Connection header received
0923 17:33:59.553783 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:59.553789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:59.553964 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:59.554138 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:59.554834 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.555572 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.557578 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.558040 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.558151 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:59.558866 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:59.559434 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:59.559555 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:59.559771 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 17:33:59.559859 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13581,"thread_start_us":13463,"threads_started":1}
I20260923 17:33:59.583575 31445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:59.566301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52884 (local address 127.8.29.254:38117)
0923 17:33:59.574070 (+ 7769us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:59.574076 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:59.574098 (+ 22us) server_negotiation.cc:407] Connection header received
0923 17:33:59.574219 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:59.574225 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:59.574435 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:59.574645 (+ 210us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:59.575433 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.576564 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.579124 (+ 2560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.579838 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.579982 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:59.582419 (+ 2437us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:59.582896 (+ 477us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:59.583002 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:59.583171 (+ 169us) server_negotiation.cc:299] Negotiation successful
0923 17:33:59.583263 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7638,"thread_start_us":7543,"threads_started":1}
I20260923 17:33:59.586086 31438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:59.566362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46664 (local address 127.8.29.252:38887)
0923 17:33:59.566603 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:59.566609 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:59.579156 (+ 12547us) server_negotiation.cc:407] Connection header received
0923 17:33:59.579405 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:59.579411 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:59.579496 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:59.579552 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:59.580270 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.581311 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.583987 (+ 2676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.584575 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.584671 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:59.585302 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:59.585436 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:59.585498 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:59.585721 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 17:33:59.585820 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260923 17:33:59.588342 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 31087
I20260923 17:33:59.589350 31149 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:59.589396 31432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 17:33:59.566027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.8.29.253:34693)
0923 17:33:59.566211 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0923 17:33:59.566217 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 17:33:59.580823 (+ 14606us) server_negotiation.cc:407] Connection header received
0923 17:33:59.581100 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 17:33:59.581106 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 17:33:59.581209 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 17:33:59.581275 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 17:33:59.583973 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.585015 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.587271 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 17:33:59.587870 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 17:33:59.587958 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 17:33:59.588589 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 17:33:59.588749 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 17:33:59.588835 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 17:33:59.589003 (+ 168us) server_negotiation.cc:299] Negotiation successful
0923 17:33:59.589096 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260923 17:33:59.729619 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 31087
I20260923 17:33:59.747702 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 31161
I20260923 17:33:59.748889 31223 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:59.862176 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 31161
I20260923 17:33:59.883318 8311 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task6Gj89p/build/asan/bin/kudu pid 31235
I20260923 17:33:59.884454 31297 generic_service.cc:196] Checking for leaks (request via RPC)
I20260923 17:33:59.991026 8311 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task6Gj89p/build/asan/bin/kudu with pid 31235
2026-09-23T17:34:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4835 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (9645 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (361481 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST